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ABSTRACT 

This briefing report on program evaluation notes that 
the new wave of family-centered, integrated services is forcing the 
field to rethink its methodology, and that the move towards 
outcome-driven reforms requires carefully defined outcomes and ways 
to measure them. Highlights of a seminar meeting held to explore 
these issues are included. The background briefing report itself 
begins with an introduction that stresses the need to reform patterns 
of delivering services to children and families. It asserts that 
creative approaches to evaluation must be developed and tested, and 
that improved and integrated data systems must be an integral part of 
improving service delivery. This report presents a broad framework 
for understanding the challenges involved with this reform. Its 
purpose is to share some preliminary thinking about what issues are, 
and about the kinds of data systems and evaluation studies needed to 
implement and assess new initiatives. Part I centers on program 
evaluation and the search for a new paradigm. It describes different 
evaluation approaches (micros imulat i on , experimentation and 
quasi-experimentation, qualitative evaluation) and their relevance to 
the new paradigm, and considers the additional approach of case 
studies. Part II discusses outcomes definition and measurement. Part 
III looks at information systems and data integration. Part IV 
considers the role of the federal government in data integration and 
evaluation. Part V presents examples of current research, data 
integration, and evaluation initiatives. Lists of guides and 
references are included. (NB) 
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DATA INTEGRATION AND EVALUATION: 

ESSENTIAL COMPONENTS OF 
FAMILY-CENTERED SYSTEMS REFORM 



Highlights of the seminar meeting held on September 17, 1993, Hart Senate Building, Room 708. 
(A supplement to the Background Priefmg Report.) 



The moderator, Theodora Ooms, introduced the seminar by noting the growing interest in 
program evaluation. The new wave of complex, family-centered, more integrated services is 
forcing the field to rethink its methodology and emphasizes the importance of improving basic data 
systems. In addition, the move towards outcome-driven reforms described in Vice President 
Gore*s Reinventing Government report (National Performance Review, 1993) spotlights the need 
to more carefully define what we mean by outcomes and how they can be measured. 

Ooms asked Heather Weiss, director of the Harvard Family Research Project and author/editor 
of several volumes on the subject, to address some of the challenges involved in the evaluation of 
family-centered, integrated systems refonn. 

Weiss said her remarics were based on the Project's close monitoring of the evolution of these 
kinds of programs in a number of states over a decade. In the course of these studies they have 
come to realize that programs by themselves are not enough, systems change is essential. "My 
colleagues and I have indeed bought into what has gotten to become the 'mantra'—faraiiy-focused, 
comprehensive, integrated, consumer-friendly systems of services—which, if taken seriously, has 
profound implications for rethinking evaluation of these initiatives." 

Underlying all her comments, Weiss said, is the premise that our usual ways of proceeding in 
evaluation are not likely to work: we are going to have to do things very differently. But first she 
began by defining the key elements of the "mantra." Family-focused services are those that look 
beyond parenting education for mothers and parent-child interaction, and focus on improving the 
interactions between members of the whole family system. Integrated means that, at a minimum, 
two people or agencies that have not worked together in the past are now working together so that 
they can begin to put together these family-focused services. Consumer friendliness is more 
difficult to defme. It has been talked about more in the business world, but no one made great 
strides in operationalizing this term. 

Defining the mantra operationally poses tremendous challenges to the field of evaluation, Weiss 
said This is because "we are inventing the plane as we fly it." These programs are not a 
bounded, prescribed treatment that allows us to say whedier "it" works or not These initiatives 
are extraordinarily complex and fluid and, for example, involve changes at multiple levels: the 
participants, frontline workers, die organization, between agencies, and in die community. Weiss 
said she wanted to present some ideas about strategies for evaluation thai make sense, the key 
ingredients of these strategies, and some of the chidlenges they pose. 

To fiist provide a concrete picture of what these programs are like, she briefly sketched the 
elements of the Gainesville Family Services Center project In this project multiple services 
(health, mental health, parent education, social ser\4ces) arc co-located in two trailers situated 
between two elementai> schools. An important parallel result was the development of a whole 



network of relationships of informal support among the parents. An extraordinary degree of 
cooperation and negotiation among many different programs and layers in the public and private 
sectors was needed to put these services in place and continue to nurture them. Treatments are 
individualized for each client and provided with varying degrees of intensity, according to the 
Deeds and characteristics of different families. This makes it extremely difficult to use traditional 
methods of evaluation — for example, it is incredibly difficult to define the "treatment." 

Since almost every player in the system is being asked to take risks and do things differently, new 
kinds of non-threatening partnerships are needed between those delivering and those evaluating 
these services. Weiss ai^ued that to assure accountability in tiiese system reforms, we have to 
lecognize there are multiple stakeholders— not only those working in the programs, but also the 
(;ustomers, frontline workers, managers, taxpayers, and policymakers. 



Principles 

Weiss then enumerated some of the key principles that are needed to guide this new ^proach to 
evaluation. 

♦ Evaluators need to develop means for providing continuous feedback to help the programs 
internalize a process of assessment, reflection, and mid-course correction. This is as simple as 
"Are you serving the people you said you were going to serve? Why aren't the other families 
that you had hoped to serve coming through the door? And it is as complex as finding ways to 
merge data systems." 

♦ From the very beginning there has to be a focus on measurable outcomes for families and 
children. And who is at the table to deteimine these outcomes is particularly important 
However, it doesn't make sense to develop long-term goals and arfter 10 years determine that 
the program did not work. There is a need to develop markers (interim, short-term indicators) 
that show whether programs are on the right track. At the front end these may be as simple as 
tlie number of conunissioners who come to the meetings. 

♦ We have to link evaluation to accountability. Evaluation in the past "has been like the tree thai 
drops in the forest and nobody knows or cares." By contrast, Weiss argued, the evaluation 
should become an integral part of the initiative. If we are really talking about outcome-driven 
assessment, it means building public accountability for those evaluations. If the outcomes are 
not good, somebody will care and do something about it This also means that we no longer 
have the outside, independent evaluator coming in and dictating what the outcomes are. We 
need a kind of participatory evaluation, particularly at the systems and community level, which 
cannot be insulated from politics. All stakeholders have to recognize and figure out how that 
can be managed. 

What kinds of resources are needed to implement these principles? Weiss said this kind of 
s^proach requires sustained internal evaluation capacity and probably some help from externa! 
evaluators. It requires raultidisciplinary teams knowledgeable not only about child and family 
outcomes but also about organizational development and systems change. And it requires keeping 
evaluation expenses reasonable enough so that the initiative can keep on its own inquiry even after 
the "formal"' evaluation has ended. In fact, Weiss emphasized that evaluation activity ^ould never 
end. '*We should leave programs with the capacity to continue to evaluate themselves if we are 
serious about converting to outcome-driven services." 



Challenges 

Weiss noted that in developing and testing this kind of approach and detemining its feasibility, we 
need to address the considerable problems faced in carr^g it out Past evaluation strategies, 
particularly with large scale evaluations, have been to bring in someone from the outside to assess 
impact using, ideally, experimental designs and random assignment We have to look 
systematically and critically at this traditional evaluation strategy and ask whether, in these new 
initiatives, ii will yield objective, credible, and relevant data. 

We also have to figure out what are relevant and useful intermediate markers of success. And who 
sits at the table and makes that determination is critically important to the likelihood of achieving 
those markers. "So it isn't something that I as the great evaluator from Harvard come in and teU 
your community/ Weiss said. ''It is something that I as a more humble evaluator can come in and 
help you figure out for yourselves with some outside expertise and notions about how other 
communities have approached the same thing, and help you spell out your own set of maikers for 
which you would then be willing to be held accountable," 

Weiss said "this mantra is now going full force and if it is going to be anything other than rhetoric 
we are going to have to figure out how we hold ourselves accountable and what we want to be held 
accountable for." What are the outcomes and when do we expect to see them? It means some 
hard work figuring out how to measure everything from changes in internal family functioning— 
which we are not so good at yet — ^to figuring out how to look at changes in overall community 
rates for abuse and neglect. 

Weiss added that it is important to figure out how to make better use of existijig information. 
There is a whole lot of information being developed in communities and states, both formally and 
informally, which we do not make good use of Weiss noted how discormected communities 
evaluating mantra-type programs are from each other. Everybody is reinventing evaluation, and 
there are very few resources for folks in the communities to access some of the hard lessons 
learned in past evaluation efforts. Evaluators must recognize that this has implications for 
networking and knowledge development and dissemination. 

Weiss concluded by reemphasi/Jng the need to reframe the way we do evaluation. It is going to be 
more intensive and it is not going to be done suictly by outsiders. It is also means figuring out 
how to make better and quicker use of information that is currentiy being developed. 



The second panelist was Elyse Kaye, vice president of James Bell Associates (JB A), a company 
that is involved in a wide range of program evaluations, including several integrated services 
projects. She began by saying that human services programs are changing radically. As a result, 
there is a tremendous need for an increased focus in the early c^ages of these irutiatives on the 
formative (i.e., process) evaluation. Later on there is definitely a need to focus on die outcomes of 
these programs, but programs are seldom sufficienUy defined at that beginning point to know what 
the outcome variables are going to be. 

Kaye noted three areas that people need to examine carefully as they are doing process evaluations: 
the (M^anization and plaiming of services integration; service delivery issues; and information 
system supports. 

Organization and control. When organizing and plarming service integration projects, 
evaluators need to look carefully at who controls the project JBA evaluated the five services 
integration pilot projects funded by the U.S. Department of Health and Human Services in the mid- 
1980s and a key finding was tiiat a lead agency always got die grant. The question becomes who 
is ihe lead agency and what requirements were put on them for distributing money, and what 
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incentives were there to truly bring other agencies into the process. In some of the projects the 
money went to the human services depanment, stayed within thai department, and services 
integration occuned within the department. But people did not reach across agencies and funds 
didn't flow out of the lead agency in a way that would bring other people in. 

One exception to this was in South Carolina, where the funding went to a legislative reorganization 
commission. The commission set up a process of competitive bidding in the counties, open to all 
local entities, and required that at least two entities within a county submit a ^)ecific proposal. As 
a result, there was a wide range of activities and stakeholders involved and a totally different level 
of activity and innovation was attempted. 

There is also a continual tension in all evaluations on state versus local control. There's a general 
consensus that things work best when they are implememed at the local/commimity leveL This is 
not to say that there cannot be state involvement But, clearly, it must be implemented at the 
commtmity level, community people must be involved, and they must agree to sign on to the 
initiative at a very early stage. 

An issue that is often overlooked is the process of identifying the stakeholders. Stakeholders are 
defined here as anybody that either affects the initiative or is affected by the initiative, including all 
levels of state agency staff, political people, private agencies, businesses, local agencies, and 
unions. It means all levels of staff witiiin an agency and, most importantly, tiie families who need 
services. 

Past evaltiations have often focused on the importance of high-level political commitment to a 
service integration initiative. But according to Kaye, they have found that while high-level political 
commitment is vital in initiating the service integration efforts, it is not necessarily the most 
important thing in sustaining and implementing them. "We jokingly say that sustained political 
commitment is a contradiction in terras!" Therefore, you had better involve middle management 
fmm the agencies in the process while you have that political commitment 

One of the most important questions to ask to people involved in plarming and organizing services 
integration efforts is, 'To whom do you report about what you arc doing in this initiative?" And 
people ask, "To whom do I report in general or to whom do I report about this comnuttee that is 
trying to put together the common assessment form?" There is a need to track not only how people 
talk to each otter and to agencies tiiey work witii, but also how all these people feed the 
information back into their own agencies and to colleagues. 

Kaye noted that the most common observance of staff when conducting case studies of these 
programs is that "nobody included us in this effort" or "nobody consulted with us." This is not 
always quite accurate, somebody was consulted, but maybe the wrong people. There are 
ccHnmittees that talk to other committees, but there is not a process in place to communicate with 
the rest of the staff. 

Service delivery issues. A central concern in past and current initiatives is how to effectively 
deliver multiple services. Under the Service Integration Pilot Projects, JB A observed five states 
trying to implement some kind of management system. A couple of sites essentially tried to graft 
their case management system onto existijig workers, taking social services staff and public health 
nurses and training them to be case managers. Others starteid from scratch and hired and trained 
staff from a variety of backgrounds. In general, it was very difficult to graft case management to 
an existing unit that already had a professional identity. Kaye added that there needs to be a focus 
on starting a new unit of case management as a professional activity and not as something that is 
added to an existing job description. 



The concept of co-location of services is becoming another popular element in these initiatives, 
Kaye said. And while it has become a very handy buzzword, people have totally different 
concepts about what co-location means. There are potentially three reasons for doing co-location 
of services. First, to faciliute access for clients. Second, to promote coordinated case planning. 
In theory, if staff are in the same place it v;ill be easier for them to sit down and talk to one ano&er 
and make decisions about how to proceed ii serving a family. The third reason is to reduce 
administrative costs. There should be some economies of scale by limiting the number of people 
that have to do home visits, combining some activiti^, and so on. 

If we arc to measure the success of co-location, it is clearly important to clarify what the goals are 
and what steps are being taken to implement them, as they will not be achieved automatically. 
Kaye asserted that even if the services arc located in one building, without a system to coordinate 
appointments, people will still have to return several times because their appointments are all at 
different times on different days. 

Similarly, coordinated case planning is increasingly being used. There is a difference between this 
and simply out-stationing staff (where someone still reports back to the headquarters but spends 
some afternoons at a local site). Out-stationing may be a particular good thing in rural areas, but 
unless there is some case management structure at this co-located site — where people report 
through that structure and where there is case supervision, case conferences, etc. — this is not 
coondinaied case planning. 

Another important issue is that money in these initiatives is often provided for integration, not for 
delivering services. However, Kaye believes it is virtually impossible to improve family outcomes 
if one expands outreach efforts and make access easier but then does nothing to make more 
services available. 

Integrated data bases. In her judgement, to date there arf. no truly integrated systems up and 
running. Many promising efforts are currently underway, but there have also been a lot of false 
starts and problems. This is true for both the large interagency databases where people are trying 
to get records from health departments or income assistance programs, or school records, as well 
as programs for family center workers trying to implement a good case management tracking 
system. Some of the barriers we have noted, Kaye said, include the fact that human services are 
woefully lacking in the needed technology— poor choices in software, a lack of appropriate 
hardware, and insufficient programming experience. Another major barrier, in case management 
tracking systems in particular, is a lack of computer terminals that the workers can use. 

Other barriers include the often noted privacy and confidentiality issues which Kaye believes have 
been somewhat exaggerated. To some extent, confidentiality is often used as an excuse. "My data 
is more confidential than your data." In fact, if you look closely at the laws, there are ways that 
confidentiality can be assured and the data can still be shared. However, strong feelings arc 
involved and there is sometimes a need to separate what are actual legal constraints from the 
opinions of the people involved. 

A bigger problem is that the development of most systems has not been based on the needs of 
iriformation management We do not start out by asking, "What's the output rcport? What is the 
document that you would like to see on a monthly, quarterly, annual basis that will help you by 
telling you how your program is woridng? Instead, we developed a l)lack hole' — everybody 
starts with the d^ input form. And the result is that we have all lands of information falling into a 
system. Staff stop filling out the forms because they get no feedback because no one has figured 
out how to get the output in an organized format which is useful to feed back lo people." Kaye 
said that these are some of the issues they feel are most important for evaluators to work with sites 
to improve on. Then, as we are better able to solve these problems, we can focus on outcome 
measures that arc meaningful to all parties. 
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The third panelist spoke as a state administrator from a state which has launched a major reform 
initiative and thus has the "insider's" responsibility for deciding what kind of evaluation will be 
done and how to implement it. Donna Stark, the state director of Maryland's Children and 
Family Services Reform Initiative, introduced herself as "the lunatic who's trying to fly the plane 
that has one wing, incomplete wiring, little plumbing, and we are fljong in a thunderstorm! As I 
heard Heather Weiss's meuphor, I reacted, yes, ind^ this is what it feels like!" 

Staik began by saying the most important question posed to her early on by the Governor after she 
had outlSied to him the key elements of the *'mantra" was, ''Will you please tell me what this will 
do for Maryland's families ?" And this is the central point: first and foremost, we have to ask 
ourselves if the reform is going to result in positive outcomes for families. 

As Stark briefly suinmarized, the Maryland initiative is attempting to change the system so that 
families can get timely, comprehensive services in their homes and communities. That couldn't be 
done without changing the entire infrastructure of decisionmaking, funding, and accountability, 
and that is where a lot of time is being spent Stark said they want to change the system so that 
decisions can be made on a collaborative basis at the neighborhood, city, county, and state levels. 
To do this, structures such as the Governor's sub-cabinet on children, youth, and families, local 
governing boards, and community boards have been established. They are looking at a financing 
strategy of redirecting money from high ends of the system — placement dollars into prevention 
dollars— refinancing activities that will maximi2e federal fimding opportunities, reinvesting those 
dollars into child and family sen/ices, and rewarding communities, through incentives, for cost- 
effectiveness and achieving outcomes. 

Stark said they have had a great deal of success, but still have a very, very long way to go. There 
are lots of prototype programs in the state that are making a difference in families' Uves, but one of 
the challenges is to bring them to scale. 

Self-evaluation strategy. In terms of today's topic, however. Stark said what they have 
learned most is the process of "self-evaluation" (as developed and designed by their consultant 
evaluator and "guru," Dr. Lynn Usher from the University of North Carolina). External 
evaluations and process evaluations are very important, but they are very different from the self- 
evaluation process. Before you can begin to plan an external evaluation, the first thing programs 
need to have sufficient integrity so that they can be evaluated. 

Self-evaluation, Stark explained, is a continuing, constant process and has really become the 
driving force in Maryland. Worlcers, supervisors, and legislators have continuous data, based on 
outcomes, to help them examine what they are doing, see if it is what they intended to do, and 
make mid-course corrections. "The self-evaluation strategy gives us this opportunity. Most 
importantly, it gives us feedback for management and policy decisions all along the way. It's not 
anecdotal or gut feelings, but looking at the data in a timely way to develop and change policies 
which enable us to improve services for families." 

Staik then outlined some of the first steps they had to take to Implement this strategy. The 
Governor's subcabinet was created by law to take responsibility for the development of 
noncategorical, interagency services. They didn't really have any idea where to start Everybody 
had an agenda, but they didn't have any cross-systems data whatsoever. For example, the child 
welfare agency really had no idea about how the juvenile justice agency was spending its money. 
They began by collecting crosji-systems data and preparing an integrated children and family 
services budget They started listing the services provided, but were immediately challenged by a 
lack of common definitions. Does an out-of-home placement mean the same for child welfare, and 
the juvenUe justice, and mental health agencies? It was important to agree upon common 
definitions so they could begin to create characters and fields in their data system that made some 
sense. 
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They also developed a five-year trend analysis which looked at the flow of children out of their 
families and back into their homes and communities. The data was so revealing: one-third of the 
entire state human services budget was being spent on out-of-home placement They had 
articulated a value — that the best place for children to be is with their families—that was 
inconsistent with the amount of doUars spent on cut-of-horae services. As a result, the first 
decision of the subcabinet was to decide on joint priorities and develop joint responsibility for 
solving and addressing these problems. 

The Reform Initiative's agenda was shaped by these data. Whether you came from a perspective 
of families or from the perspective of economics, there was no question that this had to be a 
priority. Developing the elements of the mantra was easy, but developing the capacity for feedback 
about their success at enabling more families to stay together in a healthy and safe way was much 
harder. 

The HomeBase data management system. They developed a set of questions, but did not 
know how they were going to collect the data to answer these questions. Over time they developed 
a software package called HomeBase, a case management tool that provides the data capacity to 
evaluate their self -evaluation questions. Woikers can sit at the computer and ask if they are using 
the most appropriate model. Whether it is a six-week intensive service or a 12-week intensive 
service with step-down, the worker can query the system and get a print out that shows what 
services are being provided and where and how the worker's time is being spent It is an 
immediate feedback loop to the woiker and has implications for supervision and management The 
aggregation of data gives local governing entities, local authorities, and the stale subcabinet the 
information needed to determine whether the initiative is doing what it intended and to make mid- 
course corrections in a very timely way. For example, when they first used it in Prince George's 
County, they were astounded to discover they they were not doing what they intended to do, and 
this data helped them restructure their plans. 

The HomeB'i*' ^ system is a family record, not an individual record, and this is an innovative aspect 
of the evalt . process. The system can be used to look at all the people in the family 
constellation, as defined by the family, see if there are any other members of the family being 
served by other agencies, and track family progress toward goal achievement. 

HomeBase has been developed by family workers and is very user friendly. It has the capacity to 
track goals and strategies which are logically associated with outcomes. It also has the capacity to 
aggregate data that helps answer more questions and is used as feedback for planning and 
management purposes at the state, local, and unit level. Although they are very pleased with it, the 
HomeBase system is a long way from being the statewide integrated management information 
system it has the potential to become. 

According to Stark, in Maryland they continue to face several challenges and questions. 

• With the input of caseworkers themselves, they are developing their own family assessment 
form to determine basic demographics and characteristics, strengths, and needs of families. 
This form will be folded into the HomeBase system and used by all the caseworkers across the 
state. 

• They continue to struggle with the development of an integrated MIS system^ One major 
problem is that federal and state information system doUars are dedicated to the billing, 
auditing, and case management needs of single state agencies, not to the development of an 
integrated state systems. Maryland also has very substantial prior investments in mainframes 
(200 million dollars have gone into the child welfare MIS system) and integrating several 
systems is very challenging. 
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Is the confidentiality issue real or a red herring? There clearly is resistance to sharing data. One of 
the issues they've come across is protecting the access to one's data because it can be manipulated 
to achieve desired results. And there is a genuine concern that it may reveal some program deficits 
that would preferably be dealt with internally rather than publicly. 

In closing, Stark offered a set of brief recommendations to those who influence legislation, 
language, policy, and funding that goes towards the development of outcome-based systems and 
the development of the data capacity to look at outcome-based systems. 

• Whenever possible, funds should be allocated for the development of an integrated system or 
integrating current systems, rather than to a single agency information system. 

• Funding should flow through collaborative bodies like Maryland's subcabinet because if you 
give funds to one agency, that's where it stays, even it is for a collaborative process. 
Proposals for information systems should come from that collaborative body» who will then be 
held accountable for the data and outcomes. 

• V/e must develop a system that focuses on outcomes for families, not bureaucratic or budget- 
based outcomes, but outcomes that measure the difference made for families and the 
community. 

• Self-evaluation strategies should be developed and external evaluations shouldn't be conducu*'i 
until you have assurances that you are measuring what you intended to measure and 
implement 

Stark concluded by saying that it is important for projects to network with each other. "It is cnizy 
for everyone to woric separately when, if we have the capacity to share information and work as a 
collaborative, we could really do something that is much more cost-effective than our current 
strategies." 



Points Raised During Discussion 

• A participant from Wider Opportunities for Women asked about the benefits and pitfalls of 
involving clients in the self-evaluation process. 

Stark responded that the city of Baltimore has developed a number of processes, including a 
client satisfaction survey, and held a series of meetings with both adults and children to ai>k 
ihem to give a critical analysis of the sendees they received from the new system. What hasn't 
been done yet. Stark said, is to integrate that kind of process into the data set 

Weiss said that several years ago, as a result of her interest in empowerment and family 
support, she and a colleague wrote a paper on participatory evaluation strategies. They were 
struck wiUi the paradox Uiat the programs were talking about empowerment but were using 
basically disempowering evaluation strategies. Weiss added that the paper raised the need but 
didn't ^ve a lot of specifics about how to develop useful and credible participatory strate^es 
(Weiss and Greene, 1992). True participatory evaluations include client input in determining 
what the questions are, possibly client participation in collecting the data, but certainly client 
participation in helping in the analysis and interpretation of what that data means and in some of 
the problem solving related to the inevitable mid-course corrections. 

Kaye added tiiat studies typically focus on family satisfaction with services. While important, 
ahnost any program with a soci^ service base that has trained staff reasonably well to be 
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sensitive and positive in their work with families will yield very high scores on satisfaction 
frora families. So it is not always the best tool for discerning small differences or learning 
more about which specifics of a program are helpful and which are not According to Kaye, in 
order to get a better sense of what happened rather than sticking with the questions about 
satisfaction, some surveys are starting to ask more specific things from families about what 
their goals are and who set the goals. Kaye suggested moving more towards focus group 
formats and other formats that may be more conducive to a broader discussion of serWces and 
family interactions with the system. 

Ooms added that we are struggling to take the customer-driven model that is very much the 
thinking in the commercial sector and refrarae it for social policy. This has never really been 
done before, except in asking about satisfaction with services received. 

Another questioner asked if the HoraeBase system in Maryland can be asked a question, "Am I 
using the right model?" Is that a decision support system? If not, are there any other programs 
that are using decision support systems? 

Virginia Ives, who directed the development of HomeBase, responded that it is not a decision 
support system per se, that is: if x happens what should I do? ACTLOG, one aspect of 
HomeBase, has more to do with how the case managers are spending their time and it analyzes 
that information and draws a picture of it in a graphic form. As a result, an image of what the 
kinds of intensive family-focused services should look like over a course of time in the first, 
second, third, fourth, or sixth week of service is produced. This product could be evolved to 
include more decision support capacities. 

A participant frora the National Council on Family Relations asked to what extent the outcomes 
for families had been defmed. She noted that we talk a lot about dysfunctional families, but 
asked if the panelists had decided what they would call a functional family. Many times 
participants respond that they liked the programs and were satisfied, but when you look at 
behaviors and measure outcomes, not much had changed. 

Stark answered that in Maryland it means a family making decisions about what kinds of 
outcomes they want for themselves. They have some cross-systems outcomes: more children 
remaining with their families, and they can count the numbers of kids. When it comes to 
outcomes for families, it is the family who will decide they want their child to be in school five 
days a week. In Maryland they try to have the family quantify for themselves where they are 
and where they would like to be. This automated system allows the family, worker, 
supervisor, manager, and state to track progress towards the family's desired outcomes. 

The questioner responded that she recognized the validity of helping people make their own 
decisions and carry them out, but there are other problems that the family may not recognize or 
be dealing with that the case worker or community may feel need to be addressed, for example, 
dealing with alcoholism or spouse abuse. How are these issues dealt with? 

Kaye said that evaluations need data for all different kinds of purposes and levels of 
stdceholders, including legislatures or other funding sources. Both she, and later Weiss, 
agreed that there is a need to track certain kinds of key measures that everybody can understand 
and are easily accessible, and that translate into taxpayer dollars such as, out-of-home 
placements, child abuse and neglect, truancy, and delinquency reports. But there is also a need 
to focus on learning how to measure softer issues of intrafamily behavior that may translate 
later into these child outcome behaviors. Weiss worried as a researcher that if we only track 
broad indicator measures, we will lose some of those microunderstandings that enable us to 
comprehend why some of these complex and subtle interventions work. 




A questioner commented that the panelists advocated inexpensive evaluation so that it can be 
ongoiijg, but with cutbacks in funding and cuts in evaluation, how do you explain to people 
that it is important? 

Weiss replied that if you are going to say to people that they are going to be held accountable 
for outcomes, you have to give them the apparatus that enables them to have that 
accountability. If we are going to require outcome-driven systems, we have to give people the 
evaluation capacity to produce, reflect on, and demonstrate that they have achieved the 
outcomes. It isn't going to be easy and probably won't be cheap. 

A participant from NASBE asked to what extent evaluation designs are influenced, positively 
or negatively, by federal and state reporting requirements and confidentiality laws. 

Stait replied that both the reality and the myth around confidentiality is challenging. There is 
the challenge of protecting people's rights to privacy, which she aclmowledged is very 
important. ''But then how do we get through that to look at who is or isn't being served. We 
are told we can't develop this kind of (shared) system because there are all these separate 
reporting requirements for the entitlement progr^—IV-B, IV-E, and Medicaid-- so we have 
to develop our own (separate) systems to be responsive to what the federal government is 
asking for. This is a maze that clearly needs Xp be unraveled. "In fact, federal money is being 
provided right now to get state child welfare systems to set up new, separate data information 
systems such as the Adoption and Foster Analysis and Reporting System (AFCARS). 
How does this all fit together? When are vft going to integrate these requests? 

Ooras concluded from Stark's comments that it would be clearly useful if the Feds got together 
and coordinated their reporting requirements. 

Kaye said she also thought there is a need to develop confidentiality rules that take into account 
the need for legitimate research purposes. "We woric with many states where there are clear 
precedures to ensure tjiat data are carefully treated and that confidentiality is observed, but 
where it is relatively a little bit saner in terms of getting access to the data and where there is 
recognition of the need for data for research and evaluation purposes." 

A participant from the National Center for Service Integration wanted to add to Donna Staik's 
comment about the need for states and localities struggling to develop integrated data systems 
to talk to one another. She noted that the Center had just received a grant from the Ford 
Foundation to do a survey to find out about integrated data systems teing developed at 
different levels around the country and examine the challenges involved in that process. 

Ooms commented that the first step clearly was a descriptive inventory such as this of what is 
going on out there. The next step some foundations and others are interested in facilitating is 
peer-to-peer technical assistance on these issues. There just aren't enough experts like these at 
the table to meet the need. And often you learn more from yovi peers in another state or 
community. 
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Data Integration and Evaluation: 

Essential Components of 
Family-Centered Systems Reform 

Background Briefing Report 



INTRODUCTION 

There is a growing sense of urgency about the need to reform our present patterns of delivering 
services to children and fainilies. The guiding concept of the reform movement—that services 
need to be outcome-driven — is strongly echoed in the recent report of Vice President Gore's 
National Performance Review which asserts that government must be outcome based and customer 
driven. "Management in government does not judge most programs by whether they work or not. 
Instead, government typically measures program activity — how much it spends on them, or how 
many people it has assigned to staff them*" (Gore, 1993: 72). Judging by results, the present child 
and family services system is seriously failing its customers. 

In order to achieve better outcomes^ major changes in the design and delivery of family and child 
service systems are clearly necessary. Services must become more comprehensive, integrated, 
family centered, and consumer friendly. Many initiatives are underway and others are being 
planned to implement these system goals. However, the new vision of a reformed system is 
currently far ahead of capacity. We do not yet know how to implement the changes—although we 
are begiiming to learn. Nor do we yet know whether, once implemented, these changes will 
improve outcomes — although there is some promising evidence that they can. Data and evaluation 
must play a critical role in building capacity for systems change, assuring accountability, and 
transforming the bold rhetoric into reality. 

Political pressures for greater accountability, combined with new information technology, create an 
opportunity to reexamine the role of evaluation in the evolving field of child and family services. 
Initial attempts to evaluate the new integrated service and systems reform initiatives reveal the 
complexity and magnitude of the challenge and difficulties of doing so using traditional 
approaches. Two propositions are gaining increasing support 

(i) New creative approaches to evaluation need to be developed and tested. 

(ii) Improved and integrated data systems must be viewed as an integral part of improving 
service delivery. 

At this point in time there are many more questions than answers. What this report aims to do is 
present a broad framework for understanding the nature of the difficulties and challenges involved 
with this kind of reform. (This is the fifth seminar in our series on family-centered, integrated 
services reform.) Our overall purpose is to share some of the preliminary thinking about what 
these issues are, and about the kinds of data systems and evaluation studies need^ to help 
implement and assess the new initiatives. We will specifically focus on some of the following key 
questions: 



• What are the le&sons from previous and current attempts to evaluate service integration and 
systems reform? 

• What seem to be some of the appropriate evaluation designs and methodologies? What 
outcome measures need to be used? 

• How can new information technologies (including management information systems and 
databases) be used to improve service delivery, evaluate programs, and increase 
knowledge? 

• What are some of the barriers to evaluating integrated and family-centered programs? 

• What is, and what should be, the federal role in promoting, funding, and supporting data 
systems and evaluation? 



Many Purposes of Data Collection and Evaluation 

The field of program evaluation is expanding its boundaries. As the revolution in child and family 
services proceeds, it is beginning to be understood that different audiences and stakeholders 
require different types of child and family data and program-related information at different stages. 
As an introduction to our discussion, we list below some of these objectives and the types of 
studies that address them, many of which will be discassed at greater length in the body of this 
report 

• To inform and alert people to a problem or need requiring some action (child and family 
indicators). Audience: media, public officials, community leaders, policymakers. 

• To understand the factors contributing to the problem or need in order to design an 
appropriate policy or program intervention (survey research, focus groups, data matches). 
Audience: researchers, program administrators, 

• To assess the extent to which existing services in a community (or state) adequately meet 
the needs or respond to the problem (community or statewide needs assessments, cross- 
system data matches). Audience: advocates, policymakers. 

• To assess/predict the probable future costs of a benefit or program change (micro- 
simulation). Audience: policymakers, taxpayers. 

• To assess the appropriate timing and design for an evaluation of a new interventiion or 
' policy reform (evaluability assessment). Audience: policymakers, program funders. 

• To document, describe, and assess the ways in which a program is being implemented in 
order to help improve the program (process evaluations). Audience: administrators, 
supervisors, frontline staff. 

• To assess the immediate outcomes and long-run impact of a program or policy intervention 
(experimental demonstration studies) on program participants, their families, other 
institutions, and the community (outcome or impact evaluations). Audience: fimders, 
administrators, advocatesy public. 

• To determine whether the benefits achieved by the new program or policy change are worth 
the increased costs (cost-benefit studies). Audience: fimders, policymakers, advocates, 
taxpayers. 




• To determine which of a number of similar programs or policies achieves the best results in 
dollar terms (cost-effectiveness). Audience: funders, policymakers, advocates, taxpayers. 

♦ To assess the current state of knowledge by conducting a systematic, scientific review of 
the published findings of existing evaluations (systematic reviews and meta analyses). 
Audience: researchers, policymakers. 



PART I. PROGRAM EVALUATION: THE SEARCH FOR A NEW PARADIGM 

(Sources: Campbell, 1987; Corbett, 1990; Heckman, 1992; Levitan, 1992; Manski, 1990; Manski 
and Garfinkel, 1992; Prosser, 1992; Weiss and Greene, 1982) 

Systematic program evaluation is a young discipline conceived in the sixties out of policymakers' 
desire to find out whether newly enacted anti-poverty programs were achieving their gois. But 
the initial question the field addressed—Does it work?-- proved to be overly simplistic. The 
purposes of evaluation have since multiplied and the stakeholders become more numerous and 
diverse. As the field has evolved, its theory and methods have broadened and become more 
complex. 

Recent publicity given to several flagship evaluations have convinced policymakers of the power 
and utility of the discipline and its capacity to deliver usefiil information. Ironically, at the same 
time, the field itself is in a state of considerable ferment and uncertainty. A number of its leaders 
are raising questions about the current reliance on randomized experimental design as the method 
of choice. And the emerging family-centered, integrated service models and system reform 
initiatives pose many new challenges that evaluators are not yet sure how to respond to. 

As essential background information to our later discussion of these special challenges, we briefly 
review the progress, issues, and controversies that have accompanied the growth and development 
of program evaluation over the past three decades. 



Background: The origins and development of the field 

The application of scientific methods of assessment to the field of social policy first began in the 
sixties when the steady expansion of social welfare programs called forth the initial efforts to 
assess whether these programs were acliiex Ing their goals. The advent of computers made it 
possible to gather and analyze large bodies of information for the first time, and the research 
community developed increasingly sophisticated methods for doing so. 

The first Congressionally maxidated evaluation of a social program, a juvenile delinquency project, 
was recorded in 1962 (Levitan, 1992). Ahnost every substantial social program since then has 
been subjected to some form of evaluation. Literally thousands of evaluations have been 
conducted, many from an academic, university base. New professional evaluation journals and 
societies were founded. The demand for evaluation became so great that a number of private 
research firms, both for profit and nonprofit, began to dominate the field. Consequendy, as 
Levitan points out, evaluation became not only a new discipline but a new industry. 

Policymakers' interest in evaluation, which has traditionally been lukewarm, is definitely 
increasing. While their rhetoric in support of evaluation far outpaces the allocation of resources to 
pay for it, most legislation funding new programs now requires some form of evaluation. For 
example, the findings of tiie state WIN demonstrations helped shape and build support for the 
Family Support Act of 1988. This legislation mandated evaluation of the JOBS program and even 
specified in unusual detail the design to be used. 
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However, one of the field's best known scholars points out that while "a consensus exists that 
program evaluation is important and should be an integral part of the policy process. 'There is no 
consensus on the manner in which evaluations should be performed and the way their findings 
should be interpreted" (Manski, 1992: 1). 



Different evaluation approaches and their relevance to the new paradigm 

As Levitan (1992) succinctly explains, three principal approaches to evaluation have emerged over 
the past three decades: (i) microsimulation, (ii) experimentation, and (iii) qualitative evaluation. 
The first two reflect the dominant ideology which has emphasized quantitative information 
coUected as part of controlled social experiments in which the primary goal is to measure the effect 
of the program intervention on the program participants. This is not surprising since evaluation's 
leading practitioriers are, for the most part, trained in the methods and analytic tools of economics, 
the most quantitative of the social sciences. However, many other types of social scientists are 
increasingly becoming involved—political scientists, psychologists, sociologists, social workers, 
and ethnographers — and there is renewed interest in the value of the qualitative approaches they 
bring to the field. 

The revolution in the way family and children's services are being designed and implemented is 
still in its very early stages. No community or state has yet achieved system-wide reform or a truly 
family-centered, integrated set of services. However, initial lessons about some of the challenges 
that evaluators of these reforms will have to encoimter can be drawn from a number of studies 
conducted to date and several that are ongoing. In this section of the report we draw heavily from 
Levitan *s formulations. Under each heading is a general discussion followed with comments (in 
italics) on some of these special challenges and issues. 



(i) Microsimulation 

In addition to wanting to know what works, policymakers and administrators frequently want 
answers to the question: What will this new program or these changes in policy cost? Widi the 
advent of national databases and advanced computer capacity, economists developed the techniques 
of microsimulation. Mathematical techniques are used in microsimulation to model the behavior of 
large numbers of individuals, households, or firms in response to changes in tax and welfare 
poUcy, program benefits, or economic conditions. Although they have come under some criticism, 
microsimulation models have played an important role in policy development and implementation, 
especially in the areas of tax and welfare reform. These models continue to be one of the 
evaluation industry's most valued tools. 

Relevance to the reform movement While policymakers are certainly very curious to know 
the additional costs of many of the proposed child and family service reforms, microsimulation is 
not likely to be a tool that will be used in the new paradigm for a very long time, if ever. It 
requires well-devebpedy quantitative databases and clear Aeoretical models which are currently not 
available, and may never be. 



(ii) Experimentation and Quasi-Experimentation 

While microamulation attempts to predict costs, evaluations concerned with the results of ^ctual 
interventions have relied heavily on the approach of social experimentation. In developing 
experimental designs, evaluators were clearly impressed with, and tried to emulate, the degree of 
rigor and control used in agricultural and other natural science experiments. 




The question, "Does it work?" is usually taken to mean, "Does the program achieve the desired 
impact on those who participate in it?" To be able to prove that the observed effects on a 
participant were due to the program, it is necessary to show what the effects would have been if he 
or she had not participated (known as the counterfactual). Since no one can measure what didn't 
happen, the next best approach is to find a individual with identical characteristics who did not 
participate and compare effects. But such identical matching is seldom feasible. Thus, the 
technique of random assignment of potential participants to treatment and control (nontreatment) 
groups was devised as being the closest approximation possible and the preferred option. 

Random assignment has often not been possible for various reasons, including high cost In these 
cases, comparison groups of individuals from a nearby, similar community, as nearly matched as 
possible, are typicdly used. Such comparison groups, or creating synthetic comparisons using 
existing data sets, are recommended as valid alternatives to random assignment (Card, 1988; 
Nightingale, 1991). Such studies are said to be quasi-experiments. Comparison groups may be 
easier to implement, but have other problems. For example, school-linked initiatives may choose 
two schools with a similar population for comparison purposes. However, a school's principal 
can have a large impact on the school, which could make d)e comparisons invalid. 

In the sixties and seventies a wave of program evaluations of early childhood and education (such 
as Head Start and Follow Through), youA employment, and other programs were conducted. 
Their findings were disappointing and, when negative, used to criticize and sometimes defund the 
program. However, most of these evaluations did not meet the tests of rigorous research to be able 
to adequately report on whether the programs were actually producing positive outcomes (Betsey 
and Hollister, 1987; Jacobs, 1988). Consequently, the initial high hopes for evaluation were 
temporarily dashed, and many programs had difficulty getting sufficient funding to perform high- 
quality evaluations. 

A second wave of studies of state and local welfare, employment, and training demonstrations, 
conducted by Manpower Demonstration Research Corporation (MDRC) and other research firms, 
were much more rigorously designed and implemented. Their somewhat encouraging findings 
helped to gain much greater recognition and respect for evaluation in the policy community. In the 
past decade the experimental design (using random assignment) has become the standard and is 
v^ddely touted as the best or, in some people's view, the only valid form of evaluation. The first 
national large-scale, multi-site evaluation using random assignment was the recently completed 
study of the Job Training Partnership Act (JTPA) conducted by the Department of Labor. Random 
assignment is a required feature of the national JOBS program evaluation and the Comprehensive 
Child Development Program Impact Evaluation. Random assignment was also required as part of 
the 10 state demonstration projects included in the House version of the Family Preservation and 
Family Support Act 1993 (part of the OBRA law), but this section was dropped in conference. 

Limitations of the experimental model. While there is general agreement that the 
experimental model has injected a welcome degree of rigor and objectivity into the field of program 
evaluation, critics point out several limitations (see, among others, Campbell, 1987; Levitan, 1992; 
Manski, 1990). For example, participant attrition and changes in the conununity or the program 
itself can introduce selection bias and feedback effects which are ignored in the model and limit the 
gcneralizability of the findings. Moreover, random assignment is often not practical because of 
strong resistance on ethical grounds by program staff or the community when services are denied 
to eligible applicants (as was the case in many sites in the JTPA evaluation). In addition, random 
assignment may also not be feasible because the intervention has so penetrated a given conmiunity 
that it is not possible to identify sufficient numbers of people to assign to a control group (the 
saturation effect). 

Importantly, experimental designs focus only on program impacts and fail to tell us what is in the 
"black box," namely, what components of the program are responsible for the observed effects. 
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Therefore, little is learned about what aspects of the prograra should be replicated, dropped, or 
improved. 

Finally, loo often there is inappropriate extrapolation from the findings of these tightly managed, 
well-staffed, and costly social experiments to the real world -^f large-scale implementation. (Jhis 
was the case with using the strongly positive results of the high-cost, very intensive Ypsilanii, MI, 
early childhood demonstration to justify increased support and expansion of the Head Start 
program.) 

Some critics of the reliance on experimental designs conclude that in the competition for scarce 
evaluation dollars, proportionately, too much money is being spent cm such cosdy experiments. 
They believe there needs to be a better balance between investment in experimental and other types 
of evaluation, such as designs focusing on process as well as outcomes and using more qualitative 
methods (see Levitan, 1992). 



Relevance to the reform movement 

Family support and resource programs. Comprehensive reviews of both flagship and fleet 
evaluations of family support and resource program conclude Aat the standard experimental 
design does not fit most of the family support and two- generational programs whose goals are so 
broad and often so vague (to "support and strengthen" familiesh and whose outcomes are 
individualized in operational terms for each client and family (Powell, 1988; Weiss and Halpem, 
1990, 1991; Weiss and Jacobs, 1988). This makes it very difficult to compare results between 
families within each program or across programs. In addition, the pathways of influence between 
parents* attitudes, knowledge, and behavior, and changes in their own and their child's behavior 
and vice versa are very unclear. It is not yet possible to develop hypo^eses that link specific 
interventions to predicted outcomes. 

These programs exemplify why it is important for evaluators to study the components and 
relationships within 0ie black box, " For example, the impact evaluation of the Child and Family 
Resource Programs documented an insignificant difference between the developmental level of the 
two groups of children compared Later, tiie ethnographic study documented that home visitors 
spent more time helping the low-income mothers than on the child development activities tiiat they 
were supposed to conduct during the visit. Therefore, the program outcomes reflected this shift in 
emphasis (Weiss and Halpem, 1988). 

Further, the programs themselves are being constantly readjusted and modified so that Ae "black 
box" of treatment changes over time. In addition, these programs are deliberately designed to 
accommodate different community needs and environments, which is another reason why cross- 
site comparisons of outcomes are not appropriate. 

Family preservation programs. Several reviews of the evaluation of family preservation 
programs have been conducted recently and the audiors have made similar comments about Ae 
difficulty of using experimental designs with this reform intervention. Indeed^ Ae first wave of 
family preservation evaluations did not employ experimental designs and Aeir findings of cost 
savings are, as a consequence, largely discounted The second wave of studies, which included 
sites in California, New Jersey, arid Illinois, did employ experimental designs. But because it is 
highly subjective, targeting programs on ' families at imminent risk of placement*' proved to be 
extremely difficult. 

One of the nation's leading evaluators, Peter Rossi, after a review of both waves of studies 
concludes that "0te best (and often the only) way to obtain unbiased net effects estimates is to 
employ randomized controlled experiments in impact assessments..Accordingly, the evaluation 
design of choice for family preservation programs is the randonuzed controlled experiment** 



(Rossi, 1992: 168), However, Rossi also recommends that the definition of success be broadened 
beyond ^e sole outcome of avoidance of placement since placement may sometimes be the 
preferred outcome. Other child and family outcome measures should be incorporated into ^se 
designs. 

As originally conceived, intensive family preservation programs ofAe Homebuilders type were 
highly discrete £md tightly designed programs, and separated from other child welfare services. 
However, they are becoming increasingly heterogeneous. And as they become gradually 
incorporated by states and local governments into a broader continuum of preventive services, as is 
the goal, it is less cmd less likely that e^rimental designs will be found to be useful and 
appropriate. 



(lii) QualiUdve Evaluation 

Qualitative studies collect information on a much smaller number of cases, but in much greater 
depth and on a wider range of variables. They gather infonnation from program participants, 
nonparticipants, service providers, administrators, and key informants in the community in 
attempts to explore and understand the hows and whys, as well as the impacts of the program. 
Qualitative approaches aie generally moie open-ended and exploratory in nature, striving to 
understand the factors and possible relationships involved rather than testing a particular 
hypothesis. The data collected is frequently described in process terms (relationships, behaviors, 
feelings) and are often not readily quantifiable or easily translated into dollar amounts. 

Qualitative studies employ a wide variety of methods developed by the so called ''softer social 
sciences" to gather information. These include telephone or in-person interviews which ask open- 
ended questions; participant, ethnographic, and structured program observations; focus group 
discussions; and videotaped observations of natural behavior or assi^ed tasks. Qualitative 
researchers may focus on a handful of intensive case studies of individuals or programs, gathering 
as much information as possible, or they may interview or observe larger nimibers. 

Qualitative studies were in use long before the advent of experimental approaches, but then went 
into disfavor because their findings were considered to be subjective and, therefore, the studies 
could not be replicated, nor were the findings believed to be generalizable. Over the past two 
decades, however, attempts at introducing greater rigor and consistency into methods of gathering 
and analyzing qualitative data have met with some success (see Patton, 1980, 1987). Since 
qualitative research, unlike the experimental studies, takes into account the contextual environment 
within which programs operate, some believe that the findings, though less rigorous, may be just 
as useful as tte larger sc^de, experimental studies, if not more so (Jacobs, 1988; Levitan, 1992). 

While qualitative studies are able to provide "thick descriptions" of context, program 
implementation, and systems outcomes, they have some drawbacks. Good qualitative evaluation 
is very expensive, it takes time, and is difficult to interpret When using qualitative information to 
do cross-site comparisons, observation guides must be very structured to get consistent 
informatiorL They end up resembling more a survey than an "ethnographic" description. When 
this is the case, evaluators have to assess the cost-effectiveness of using open-ended data collection 
methods. 

Relevance to the refonn movement Given the difficulties of pure outcome research when 
program goals are broody services are both flexible and comprehensive, and A^. impact is likely to 
be U)ng term rather tiian immediate, tiiere is a clear need for more sophisticate^ systemic 
evaluation designs which use a broad range of process and outcome measures. A wide range of 
qualitative designs and methods are employed in past and current evaluations of family support, 
family preservation, two- generation programs, and school-based integrated services. 



For example, Minnesota's Early Childhood Fondly Education program has adopted a systemic, 
five-tiered, stages approach to evaluation developed by Jacobs (1988). Only at the fifth tier is there 
an assessment of whether enduring changes in program participants, bo^ parents and children, 
have occurred The virtue of such an approach is that observed outcomes can be more readily 
connected to actual services or involvement in the program, rather than relying on statistical 
inferences about control group/experimental group differences. 

Foinoiative/process evaluations. One of the distinctions being made in the field of evaluation 
is between formative/process evaluations and outcome/impact evaluations. Formative/process 
evaluations are typically used in the early stages of a program evaluation to describe and 
understand how the program is being delivered, and to identify the factors that help or hinder 
program implementation or desired system reform— in other words, to imderstand what is 
happening in the black box." 

The controversies about the relative merits of quantitative and qualitative studies, and between 
impact (summative) and process (formative) designs, may be lessening somewhat as the field 
begins to appreciate that different methods and designs each have their value and complement one 
another (Jacobs, 1988). Indeed, many ongoing, large-scale studies, whether funded by the federal 
government or flie private sector, now employ mixed methods and designs, including both process 
and outcome studies. However, the process and outcome studies are generally conducted 
independently, in sequence, and sometimes even by different organizations (see (XDP, p. 32). 

Manski suggests that "the distinction between process and impact evaluation, albeit long-standing« 
is untenable" and arises from a failure to recognize that process is a part of treatment 
Implementing a new program (treatment) is not like trying out a new fertilizer. A program 
represents the summation of a complicated set of relationships between various levels of 
government and of the decisions and behavior of hundreds of program administrators and service 
providers. Manski explains that to date process evaluations generally describe only program 
administration, but "policy formation requires answers to counterfactual questions. We need to 
know how program outcomes would change if the rules and incentives given to state and local 
agencies and service providers were altered" (Manski, 1992: 3). 

Relevance to the reform movement* Careful study and analysis of the changes in the 
program or service system-" "unpacking the black box of treatment"— is recognized to be a major 
challenge for die reform movement. This will require new kinds of information and the 
development of new measures. For example, the community-based service integration initiatives 
and statewide system reforms include new forms of governance and altered relationships between 
levels of government which will need to be documented and assessed 

One of the underlying assumptions, which needs to be continually tested, is that services that share 
certain characteristics, such as being more comprehensive, coordinated, and integrated, will in fact 
produce better outcomes. Thus, the first challenge for evaluators is to operationalize in very 
specific terms what is meant by the desired features and to show that ^ey are plausible in relation 
to desired outcomes. 

The initial evaluation report of the federally funded, six-site, Community-Based Service Integration 
Facilitator grants pointed out that co-location of services was the single reform strategy most often 
used by the reform initiatives under study. The autiior comments titat the language of integration 
often cloaks some important differences Thus, co-location of services by itself, without any other 
changes, is not likely to make much difference to client outcomes. It is important to assess 
whether co-location is accompanied by other changes such as joint case planning, collaborative 
treatment, and integr^Hed eligibility, which are necessary to produce positive changes in outcome. 
This question will need to be carefully evaluated (Kaye, 1993). 
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However, a second central tenet is to recognize that current programs and service systems are not 
only ineffective in achieving their goals, but that they are too often, in medical terminology, 
iatrogenic — themselves producing negative effects and causing harm to clients and providers alike 
(Mcknight, 1990). Thus, at the core of the philosophy threading through the reform movement is 
that how services are delivered is as important as what kinds of services they are, Weiss and 
Greene (1982) indicate that family support programs with an empowerment orientation should 
promote empowerment-oriented evaluations that go further in promoting structural social chan<^e. 
These evaluadons involve established decisionmakers in die evaluauon process and center fhe 
evaluation within the local community context. Their suggestion is not that experimental or 
implementation studies he abandoned, but that evabdotions become more participatory and give a 
voice to the perspective of participants and/or staff. 

Kaye's report also points out that evaluations of service delivery initiatives typically fail to consider 
that the proposed solutions may have unintended consequences. Program evaluators typically 
assume 0iat new or improved services will have positive or at least neutral or no effects on client 
outcomes. However, service integration efforts may create as many problems as they solve, 
especially if implemented without sufficient resources and training. For example, consolidated 
application forms may be longer and more cumbersome and, in tiie hands of untrained workers, 
less effective and more burdensome tools for ejqyediting intake. 



Additional approaches 

In addition to the process and outcome evaluations, the emerging new initiatives conduct a number 
of other types of research and evaluative activities which often combine quantitative and qualitative 
approaches. Such as: 

• Conducting carefully planned, community-based needs assessments to help design the new 
service configuration or policy change. (New Beginnings) 

• Data matching in which existing administrative records and data systems are compared to 
reveal the totality of child and family service involvement across systems. (Oakland) 

• Evaluability assessments to determine what aspects of the reform initiative should, and can, 
be evaluated, using what kinds of research strategies. (Annie E. Casey/RTI; Turning Points 
Program in Washington, DC) 

Case Studies* The case study has long been stereotyped as a weak sibling among social science 
methods, although the best known— such as the Lynds* Middletown, Whyte's Street Comer 
Society, and Liebow's Tally's Comer—have generated more revealing insights and new 
productive lines of research and programs than most social science experiments. However, they 
cc»itinue to be used extensively in social science research and business and increasingly are being 
used to study program and poUcy innovations (Yin, 1984). Case studies are used as an 
exploratory, descriptive, and explanatory research strategy, and are particularly useful when the 
focus is on a complex new social event, institution, or phenomenon (reform initiative) ^vtv which 
the investigator has no control. The case study "allows an investigation to retain holistic and 
meaningful characteristics of real-life events — such as individual life-cycles, organizational and 
managerial processes, neighboiiiood change, international relations, and the maturation of 
industries" (Yin, p. 14). Case studies are designed to answer all the who, what^ when, where, 
why, and hows, drawing on a variety of types of information from a wide range of sources. 
Although there is no single blueprint for the case study approach, knowledge is accumulating about 
how to design and carry out case studies in a careful and rigorous manner, and approach the 
elusive and difficult issue of analysis. 

The case study method is being used quite often by those who are trying to describe and 
understand the emerging reform initiatives. For example, the Harvaixl Family Research Project is 
conducting several case studies of statewide family support program initiatives and the Kennedy 



School of Government conducted case studies of three state system reform Liitiatives (Maryland, 
Colorado, and Aricansas). One of the principal challenges in the case study and innovations 
literature is how to structure the inquiries so that comparisons can be made between cases and a 
foundation of theory can be built. 

Case studies are being used as part of broader evaluation designs. Abt Associates is conducting 
the evaluation of the Even Start Family Literacy Program. This evaluation has several components: 
(1) the construction of a database, the National Evaluation Information System; (2) an in-depth 
study of ten projects, half with randomized experimental designs, to complement the broad-based 
data with small-scale, detailed analysis of the relationships between services received and short- 
term outcomes; and (3) other local evaluation studies as desired by individual grantees (St Pierre, 
etoL, 1993). 



Additional issues and challenges 

• Internal versus external evaluations. One hotly debated topic is whether evaluations 
should be conducf^d by external, independent evaluators or by administrators and others involved 
in the program. There are clear advantages and disadvantages to both. Increasingly, program 
evaluations are conducted by both internal and external evaluators who assume different but related 
tasks. 

Campbell (1988) believes that using external evaluators does not solve the problem of objectivity 
and creates a natural enmity between themselves and the program staff. The solution he 
recommends lies in following the example of the natural and physical scientists, in which those 
who design the experiment also carry it out, but they protect against biasing the results through a 
tradition of cross-validation with other studies. Weiss and Halpem (1990) suggest that such cross- 
validation should yield rich and useful fmdings. 

Several evaluators in the reform movement are recommending approaches called "self-evaluation** 
or "participant evaluation," in which the administrators, supervisors, and frontline workers, with 
representatives of the current or potential client population, work together to design and implement 
the evaluation. 

A compromise between the two approaches is to involve the program staff and clients in the 
various phases of a formative, process evaluation, and bring in outside evaluators to help design 
and conduct the impact evaluation. This is especially useful when several sites or programs are 
involved because the outside evaluators can help to assure that some of the types of data being 
collected and measure used are uniform across sites, making comparisons possible. For example, 
for the national evaluation of the Head Stan Family Resource Centers, the Head Start Bureau, is 
providing coordination, technical assistance, and a definition of conmion elements to produce an 
integrated process evaluation. The same firm will be conducting the national impact evaluation 
(Griffith, 1993). 

Self-evaluation is also gaining acceptance for a variety of reasons, including the fact that the 
knowledge and skills required to conduct internal evaluations or to build self-evaluating systems 
are found more widely in public agencies and that it empowers policymakers and program staff by 
encouraging them to identify maricers of performance and monitor them on an ongoing basis in 
order to maintain and improve their performance. Lyrui Usher (1993) developed a framework and 
ccHitinues to provide support and technical assistance to build the self^valuation capacity of the 
Prince George's County (Maryland) Services Reform Initiative. 

• Timing. There is a general consensus that too many evaluations have been conducted 
prematurely, long before the program was properly in place and debugged. Campbell recommends 
that no impact evaluation get underway until the program declares that it is **proud" (Campbell, 
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1987). However, the design of the evaluation needs to be in place at the very beginning of the 
program. 

• Multiple evaluations. Currently, most grants require that the program be evaluated. Some 
of the agencies that are implementing programs under several grants have to conduct multiple 
evaluations, which sometimes imposes a tremendous burden on staff and consumers. An example 
is a school in California that has been very successful in obtaining money to start different 
innovations. At this moment there are about seven concurrent evaluations of the same students, 
staff, and families. 

• Political context: pressures and barriers. Social science is a normative exercise. Values 
come into play in the choice of programs to evaluate, the questions to ask^ and the manner in which 
studies are conducted. It is interesting to note, as Levitan does (1992), that programs providing 
social services to the poor have come under much greater pressure to be evaluated than public 
services and assistance provided to other sectors of society. 

It is generally understood that both the science and the industry of evaluation are highly responsive 
to the policy environment. Policymakers helped to generate and continue to support the 
development of the field of evaluation. However, the policy environment also creates numerous 
barriers. Political pressures to get results quickly have led to requirements for premature outcome 
evaluations. As Corbett notes (1992: 26), answers are wanted in the short terra, largely defined 
by political cycles, and are expected to give definitive answers. Where a slow accretion of 
knowledge and insights would be useful, summative statements of impact are demanded." 
Policymakers are often overly enamored of evaluations that produce "hard numbers" without 
asking how solid these numbers are and what they mean. 

Another problem is placing too much weight on the results of a single study. In the past, 
policymsikers and advocates have used the results of a single study to damn or praise a program 
prematurely and inappropriately. For example, the Westinghouse study report in 1%9 was used to 
prove that He^id Start did not work, whereas the study of the Perry Preschool Program in 
Ypsilanti, MI, more than two decades later, has been cited as definitive evidence that Head Start 
does woik. Evaluators lecommend judging the merits of a new program or policy approach based 
on the accumulated evidence of a number of studies conducted over a period of time. 

More recendy, a study by GAO (September 1992) compared the findings of two initiatives (the 
Part H program and the Annie E. Casey Foundation child welfare initiatives) with the Head Start 
program. It concluded that linking at-risk families with services was a more successful strategy 
than systems reform. Many people believe the comparison was inappropriate (comparing apples 
and oranges). But, importantly, it was premature to come to a conclusion on the efficacy of 
system reform. 

Finally, another major barrier is that policymakers have been quick to caU for evaluation studies, 
and arc increasingly mandating them, but have not provided sufficient dollars to conduct them. 
Happily, private foundations have increasingly picked up the tab for the evaluations of public 
programs. Neither policymakers nor the public seem to understand that quality evaluations arc 
expensive and time consuming. Too often, ti« government has insisted that each demonstration 
project they fund should conduct an evaluation, but then set unrealistic limits on the amoimt of 
funds to be spent on the evaluation, which limits its usefulness. For example, programs funded 
througij the Adolescent Family Life Office in DHHS could spend at a maximum oiSy about 3-5% 
on evaluation (which amounted to between $5,000-$ 15,000). This amount was subsequently 
raised to 10%. 




PART U. OUTCOMES DEFINITION AND MEASUREMENT 

(Sources: Brizius and CampbeU, 1991; CSSP, etai, 1993; Gallagher, 1993; Gardner, 1989, 
1991; McCroskey, 1991, 1993; Schorr, 1993; United Way, 1992) 

A call for outcome-driven programs and performance accountability for human service programs is 
increasingly heard in many different public policy arenas (public education and welfare, health, and 
public and privately funded system reform efforts). Outcome accountability or administration by 
results are seen as the measure of success for human service programs replacing (or at least 
decreasing the need for) centralized, bureaucratic micromanagement aiKi rigid rules (see Gore, 
1993). 

A focus on outcomes and on performance accountability requires a commitment to timely 
evaluation. And for family-centered, integrated reforms with an orientation toward systems change 
rather than isolated pilot projects, the evaluation of outcomes needs to address issues of data 
integration at the family, community, and systems levels. 

Clarification of terms 

Outcome-driven programs and policies are those that design and implement interventions or 
services that are focused on (or logically related to) measurable outcomes. It is believed that the 
use of outcome indicators will help focus attention on ''mission" rather than rules and that this will 
make it easier for policy makers to desist from an overemphasis on regulating policies and 
procedures. 

Outcomes are the results towards which intei-ventions, programs, or services are directed. 
Because integrated, family-centered programs are targeted at four different levels, outcomes must 
be measured at the child, family, system, and community level. Sometimes the term impact is 
used to refer to outcomes or changes in systems or in the commuruty. On other occasions the two 
words arc used interchangeably. 

Experts in other disciplines use the tenns goals and objectives. In these circles, a goal is an 
abstract and general statement of desired outcomes and an objective is a specific, empirical, 
operational statement about a desired observable outcome. For any given goal, many different 
objectives may be written. 

Outcomes can be defined as short term (mostiy related to immediate results and also called 
objectives) or long term (also known as goals, impacts, or standards). Short-term effects are vital 
to the development and management of effective human service systems. Long-term outcomes are 
those usually most relevant to policymakers. A short-term outcome measure for a family planning 
program is the number of teenagers served who begin using contraceptives. The long-term goal is 
to increase the number of teens managing to avoid unplanned pregnancies. 

Some outcomes are difficult to observe or measure, especially those that involve die work of 
several agencies. One example is the national educational goal that by the year 2000 every child 
will be ready for school. There are many ways of measuring die extent to which children are 
"ready" for school. Therefore, programs need to rely on several different indicators that suggest 
the presence of "school readiness." Indicators are features tiiat point to the probability of the 
outcome occurring (for example, immunization rates, participation in child development programs, 
assessments of home stimulation or of child skills). 

Outcomes cannot always be observed directiy and must be measured with proxy (or indirect) 
indicators. For example, a long-term outcome of family-centered programs is to "empower 
families." Empowerment is a concept that is difficult to operationalize. Therefore, a proxy 
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indicator inai is suggestive of the occurrence of the empowerment outcome may be used (i.e.> 
decreased amount of time that the caseworker needs to spend advocating for the family). 

Accountable y concepts have their roots in the area of management, including economic theories 
about incentives and business concepts about control. Applying these concepts to public areas, 
Brizius and Campbell (1989) defme performance accountability as a means of judging 
policies and programs by measuring their outcomes (or impacts) against agreed upon standards. It 
is the focus on outcomes that differentiates performance accountabSity from the more tradidonal 
ways of assessing policy or program performance. 

Program accountability. The traditional accountability methods established to assure that public 
funds are being spent for the specific purposes for wWch they were appropriated are increasingly 
being called into quesdon. Program accountability is currently used to assess programs by 
measuring their results against its pre-set goals or outcomes. The primary outcomes of interest are 
the changes that should occur if the program is working as planned (for example, if those adults 
who complete the job training programs get good jobs). Program accountability usually measures 
the outcomes for those people served by the particular program or set of programs. 

Policy accoimtability looks at impacts (or outcomes) for whole groups or classes of people. 
Policy accountability focuses on whether a complex set of state programs and actions are achieving 
their broad goals. Individual programs can work well while broader policies fail. A policy must 
be measured both as the sura of the results from individual programs and in relation to other 
issues. 

Standards are estimates of the suggested ideal outcoraes that an effective program or policy is 
expected to produce. For example, when a state sets a goal of reducing its infant mortality rate to a 
certain level in five years, it has established a standard forjudging performance of a set of 
healUi and social services policies. When it says diat the amount of child support paid by obligated 
parents will increase by at least five percent, it has set a standard for a state agency. Program 
performance standards usually specify (in quantifiable terras) changes desired in individual 
behavior or circumstances as a result of program interventions. Performance standards are 
increasingly being imposed on stales by the federal govemraent 

Benchmarks are incremental steps towards attaining die standards. Benchmarlcs are tangible 
measures, guideposts, or milestones that can be used for assessing progress towards meeting the 
standards. If die benchmarks are not achieved, it is unlikely diat the program or policy will meet 
the standard established. As much as possible, the benchmarks must also place a priority on 
measuring results (for example, adult literacy) rather than efforts (the amount of money spent on 
literacy education). An example of a benchmark for a standard of reducing infant mortality in a 
community by 25 percent in five years would be a 10 percent yearly increase in mothers obtaining 
prenatal care. 

Benchmarks can also be used at the case level to determine progress towards attaining goals. 
Chicago Project Match's family service workers and AFDC clients set up agreed standards and the 
short-term benchmarks to reach family's goals. 



Selecting outcomes, indicators, and benchmarks 

The identification of appropriate outcomes and benchmarks is one of the most important processes 
involved in designing and evaluating outcome-driven programs and policies, It is also one of the 
most difficult for some of the following reasons: 

• Outcomes must be defined at different levels for participants (children and parents, the family 
unit) and for the community and the service systems. 
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• Over-orientation to outcomes goes against the grain of many service-providing professionals. 
(For example, medical doctors never guarantee an outcome as a result of services given.) What 
service providers offer is service expertise, a process of understanding a problematic condition 
and bringing their best knowledge to bear on it--they do not offer a product Other than from 
an ethics standpoint, professionals are accountable for showing that they offer a knowledge- 
based, up-to-date, professional process (no physician's license is revoked for failing to cure a 
particul^ patient). From this perspective, it is easy to understand why human service 
providers are uncomfortable with being evaluated on the basis of their "outcomes." 

• Outcomes may vary according to the developmental stage of programs. For example, a 
demonstration or pilot program's objective is to gather more loiowledge about the problem and 
the program, even when the explicit objectives are specific, desirable social outcomes. At the 
beginning of demonstrations the outcomes may be very different than once the program is 
underway. 

• The same set of outcomes and benchmarks cannot fit all integrated, family-centered programs 
equally well. There may be a need to defuie core reform "prototypes" to be able to (tescribe 
more relevant outcomes. 

• Negotiating an agreed upon set of outcomes in collaborative programs can be difficult Staff 
from different disciplines or agencies may assume that their priorities should be primary (i.e., 
for educators, school performance is the primary outcome and for social service professionals, 
that child and family functioning is primary). 

Several groups are woricing on defming a set of core outcomes, indicators, and benchmarks. 
While these target different levels and are being developed for different purposes, tfiey have some 
common elements. 

• Several groups (CAPD, 1993; CSSP, et ai, 1993; Joining Forces, 1990; McCroskey, et ai, 
1991; McCroskey and Nelson, 1989) have begun working on developing a core list of 
outcomes for families and children (or outcomes for specific programs such as child 
welfare or school-linked services). These groups use data that are readily available and identify 
outcomes which are relatively easy to agree on. 

• Some publications (United Way, 1992) emphasize service delivery processes and 
relationships, principles, and some concrete program or service elements and define them as 
standards. Taken together, these standards are to be used as benchmarks to assess the well- 
being of a community's children and families, to gauge how needs are being met, and to 
develop a system for measuring changes. 

• Some guides (Blank and Melaville, 1993) delineate the key stages and tasks involved in the 
process of system change and itemize separate steps under each heading. The steps 
promoted as essential to the process are presented in a question fomi that can be used as 
benchmarks to asses the collaborative process. 

• Some (CASFC, 1991; Johnson, et al, 1989; Ooms and Preister, 1988) present evaluative 
criteria, stated in the form of a set of guiding principles or key elements that flesh out the 
concept of family-centered, collaborative services. Accreditation standards and evaluative 
criteria can be Jised as benchmarks or for outcome measurements. United Way (1992) 
provides a list of indicators for evaluating multi-sector collaboratives in community-based 
projects. 

• Some publications identify separate, broad components of system change and, under 
each, the particular characteristics or sub-elements of the components. These are called 
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indicators of system change. (Some good examples are Checklist 2 in Blank and Melaville, 
1993; Gardner, era/., 1991; Henderson, era/., 1986; and Snyder and Ooms, 1992.) These 
checklists are used as indicators of whether, and to what extent, the benchmarks have been 
achieved. 

Some groups (Kids Count, California Now) are developing outcome measures at the 
community level These community "score cards," with indicators on children and families 
and some social indicators, can be used as benchmarks to assess how children and families are 
faring, how agencies are performing, and whether certain outcomes are being attained or not 
The Kids Count project is a national, state-by-state effort that brings together the best available 
and most recent data to track children's educational, economic, social, and physical well-being 
among states and over time (CSSP, 1992). 



ISSUES AND CHALLENGES IN DESIGNING OUTCOME-DRIVEN POLICIES 

Aside from the complexity of defining a set of core outcomes, there are some challenges that 
policymakers need to address when designing outcome-driven policies and programs. 

• It is important that outcomes be developed in a public forum. If communities can pull together 
a cross-section of people to agree upon a set of outcomes and indicators which will assess their 
progress, wider ownership of both the process and outcomes will result Although there is no 
guarantee that line workers and consumers would agree on what the outcomes should be, 
establishing the dialogue is a great contribution. Issues about the specific use of the outcome 
data and the consequences attached to success or failure in achieving the outcomes should be 
clearly spelled out during these discussions. 

• Political pressure for immediate results is a potential barrier to outcome-driven policies. To be 
successful, programs require much more than well-defined outcomes. Staff and programs 
who will be held accountable must have isal authority to implement interventions and adequate 
resources to respond to the needs of multi-problem families. Few communities have all the 
services and resources needed, which has implications for both the extent to which defined 
outcomes are realistic and the degree to which managers and staff will accept what are seen as 
new and unrealistic expectations without sufficient support behind them. 

• There are serious concerns regarding the quality and availability of data from which the 
indicators of the selected outcomes must be derived. Comparisons are usually difficult because 
the definitions of terms and measures are not standardized. And often the process of data 
collection itself is flawed. Furtiierraore, evaluation shouldn't focus on outcomes alone. The 
experimental nature of many of these projects means that process evaluation will be critically 
important to the projects themselves and the wider community that is trying to learn from these 
early innovative endeavors. 

• While have gained some information about "what worics," little is known about why certain 
interventions work, or to which specific aspect of a program this success can be attributed. 
Much finer-grained research and analysis arc needed to clarify the relationship between short- 
term outcomes (which are often the most easily measured and monitored) and long-term 
impacts. 

• Some advocates believe so strongly in the power of community-wide outcome measures that 
they are proposing that any community that has adopted outcome measures be able to waive 
any restrictions on the use of current funding. A less extreme alternative would be to eliminate 
only those categorical funding requirements in a given outcomes area (such as low birthweight. 



drop-out rates, or institutionalization of children). This proposal would differ from previous 
block grant reforms in that it would condition newly pooled funds on the development of 
outcome measures set by a community-wide process rather than relying on the political process 
to spend funds appropriately. 



Measuring Outcomes 

(Sources: Brown and Zill, 1992; Cleary, 1988; Hauser, et aL 1988; Howrigan, 1988; Kimmich, 
1993; Krauss, 1988; McCroskey and Nelson, 1989; Upshur, 1988; Walker and Croker, 1988; 
Weiss and Jacobs, 1988) 

Measurement of outcomes in integrated, family-ceniered systems reform efforts present a unique 
set of challenges for evaluators. The diversity of targets (which range from faroUy to whole 
systems), scope, source, and service models of the reform initiatives demands the development of 
innovative and relevant assessment tools. 

Furthermore, the range of outcomes to be measured (which many times include contextual 
information, processes, and interactional behaviors) require: (i) different methods to collect the 
information, for example, paper and pencil tests, observation, audio and video recordings, 
surveys, focus groups, and many others; (ii) the use of valid and reliable instruments; (iii) training 
of staff using the assessment tools; and (iv) making the assessment as "family friendly," as 
possible by using the information gathered for the improvement of service delivery and making the 
assessment a participatory experience. 

The following section v^l illustrate some of the outcomes and indicators used in family-centered, 
integrated reforms; discuss some of the issues related to the measures and methods to assess the 
indicators; and, when possible, exemplify current evaluation efforts measuring those outcomes. 



CHILDREN AND YOUTH 

For the past 20 years, definition and measurement of child and youth outcomes focused primarily 
on physical or cognitive domains, often relying on measures of intelligence and achievement. 
Other areas of chfld performance (i.e., social competence) were virtually ignored, even when the 
program was designed to influence a broader range of outcomes. For example, despite the 
mandate of Head Start to influence a broad range of outcomes, more than hdf of the effectiveness 
studies conducted during the first 20 years of the program focused primarily on cognitive 
development 

While the goal of most integrated, family-centered reforms includes improving child functioning, 
by definition these programs point to the need for greater attention to family and community 
outcomes as the primary route toward this end. Therefore, evaluation needs to emphasize the 
importance of a broader approach to the measurement of child-focused outcomes. 

The following is one example of a child-focused outcome, together with the indicators, measures, 
and some benchmarks (Improved Outcomes for Children Project [CCSP, 1993]). 

Outcome: To increase rates of healthy births 

Indicators : Lower rates of: 

• Premature births 

• Low birth weight 

♦ Late or no prenatal care 

♦ Birth to school-age mothers 
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Benchmarks: • Reduced rates of unintended births 

• Reduced rates of teenage binhs 

• Increased in proportions of pregnant woman receiving prenatal care 

• Higher rates of students succeeding at school 

• Increased proportions of adolescents and young adults who arc in good health 

• Increased proportions of parent preparation for childbirth and parenting 

• Increased proportions of parents receiving family planning services and 
information 



ADULTS 

Family-centered programs target adults in two different ways: increasing their parenting skills and 
providing services to support them as adults. (These interventions may ultimately benefit the 
children but this is not the direct goal.) Traditionally, the effectiveness of family support programs 
have focused on child outcomes. Virtually nonexistent are descriptions of measures of changes in 
the parents (Upshur, 1988). 

However, during the last few decades several developments (i.e., parent involvement in early 
education programs for young children, family literacy programs, "two-generation" self- 
sufficiency programs, etc.) have begun to incorporate measurements of impact on the adults 
(development, mental health, etc.) as well as on interactive parenting behavior. 

A critical dimension of measuring parenting outcomes is the measuring of the interaction between 
parents and child. Interaction measures vary on a number of dimensions, such as the age of the 
children, the behaviors of interest, and the resources available. 

Howrigan (1988) makes two important recommendations for parent-child interaction evaluation. 
(1) There is a need to choose instruments that allow the incorporation of information on individual 
children's characteristics and to assess the child's contributions to the interactions. (2) Evaluators 
must question the imiversal applicability of parent-child interaction styles and take seriously the 
notion that environmental constraints and culture-specific ideologies powerfully affea how parents 
can and should interact with their children. To this end, he recommends using ethnographic 
methods to document communities' child-rearing goals, expectations, and theories. 

♦The National Black Child Development Institute (NBCDI) has launched The Spirit of Excellence 
Parent Empowerment Project in Washington, DC. The goal of this project is to enhance the ability 
of parents to function as individuals and as parents within the community and in educational and 
work environments. According to their own definition of empowerment, tiiis project will be 
evaluating outcomes such as: (i) family adaptability and cohesion with the FACES instrument; (ii) 
types of assistance needed with Dunst's Family Needs Scale; (iii) employment and training 
activities with the ECCO Demonstration survey; and (iv) several others (NBCDI, 1993). 

The core outcome in two-generation programs (Smith, et aL, 1990) is to increase economic self- 
sufficiency. Some of the indicators are: increased levels of education; decreased welfare 
dependency; increased job skills/vocaiional skills; and increased levels of employment 

Some of the benchmarks for assessing an individual adult's progress towards the goal would 
include: finding appropriate child care; enrolling in an alcohol abuse program; improving literacy 
skills; enrolling in a GED class; developing work-related skills; exploring job opportunities; 
making initial appointments for a job search; and getting employment. 
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4^Standard measures do not exist for most of these indicators. The impact evaluation of the 
Comprehensive Child Development Program (CCDP) is using the following indicators to measure 
economic self-sufficiency: education certificates, degrees, employment salary, income, and use of 
government assistance. These data are gathered in the parent interview (Lopez, 1993). 

♦Child Trends is conducting the Child and Family Subgroup Study, a ccwnponent of the national 
evaluation of the JOBS program. The goals of this evaluation are to: (i) improve upon existing 
survey naeasuies of the mother-child relationships; (ii) explore the feasibility of carrying out duect 
observations of mother-child interaction within the context of a survey research study; and (iii) 
examine the congruence between self-report measures and observational measures of the mother- 
child relationship (Zaslow, et al., 1993). 



FAMILY SYSTEM 

There is a growing recognition within the services integration movement thai services must be 
more family centered, (The terms family centered, family focused, and family based are generally 
used interchangeably.) At the direct service level, ifamily centered refers to services targeted to the 
family as a unit In this area, the challenge is developing methods to assess family outcomes. 
What do we mean by family strengthening or supporting families? 

In the child welfare field most measures have focused on risk assessment (to protect the child). 
Most of the instruments focus on deficits and pathology, and generally do not assess strengths. As 
a result, some efforts are underway to improve assessment tools. 

♦The Child Welfare League of America (CWLA) is currently engaged in a project to improve the 
family assessment process used by their agencies by designing a family strengths inventory (FSI). 
The FSI will be different than measuring parenting ability. It will look at the capacities of 
individual family members, the family's support system, and the community resources that could 
be tapped to increase parenting capacity. This instrument will be able to measure outcome in the 
parenting domain (Day, 1993). 

Also in the child welfare field, McCroskey (1989) and Kimmich (1993) have developed family 
assessment forms that are useful for evaluating outcomes of in-home family support services, 
which are especially relevant to integrated systems reforms efforts. 

Clinical advancements in the family systems field have not had a large impact on the evaluation of 
family-centered programs because of the lack of family assessment instruments. However, in the 
last few years considerable work has been done and effective assessment of family functioning has 
improved dramatically in terms of reliability, validity, and clinical utility (Olson, 1993). 

Some of the instnmients most widely used in the family field to measure outcomes of marital and 
family functioning are: (1) Family Adaptability and Cohesion Evaluation Scales (FACES II) 
(Olson, et aL 1985); (2) Family Assessment Device (FAD) (Epstein, et aL 1983); (3) Family 
Assessment Measure (FAM EI) (Skinner, et aL, 1984); (4) Family Environment Scale (FES) 
(Moos, 198 1); and (5) Self Report Inventory (SRI) (Hulgus, et a/., 1985). These instruments 
measure outcomes of cohesion, adaptability, conflict resolution, values, family health, family 
strengths, and others. 
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NEIGHBORHOOD 

(Source: Smith, 1993) 

In an effort to begin developing methods and measures to be used in the evaluation of 
neighborhood-level changes, the Foundation for Child Development (PCD) recently funded five 
projects serving low-income families with young children. These projects will assess different (as 
weU as overiapping) characteristics of neighborhoods. These include the adequacy of social, 
health, and early childhood services available to families; the quality of social support provided by 
neighbors and community institutions; social climate and public behavior (e.g., criminal activity, 
nature of social transactions in public places); and the quality of housing, transportation, and 
neighbcM-hood facilities such as parks and playgrounds. A variety of data collection methods and 
instruments will be piloted in the projects: focus groups, in-person interviews with parents and 
service providers, participant observation, questionnaires completed by mail^ and documentation 
based on neighborhood observation and collection of public data. In several instances, findings 
related to a similar construct will be compared across different sources of data (e.g., ethnographic 
observations vs. parent report). Public statistics such as community demographics and rates of 
crime will also be used in some of the projects as a context for interpreting neighborfiood profiles. 

These projects will provide a laboratory for the development of a wide range of measures and 
methods relevant to the neighboAood-level assessment of family conditions and social ecology. 



SERVICE SYSTEMS 

Coordination and collaboration are core strategies of the services integration movement But they 
are broad concepts that have yet to be operationalized into standards and benchmarks. (The Family 
Impact Seminar is initiating a project to identify benchmarks for systems change.) 

Some of the specific indicators of coordination at the service delivery level are co-location of 
services and one-stop shopping initiatives; at the program level, the flexibility to use resources 
across boundaries, the joint roles of staff, and joint efforts at intake; and at the city level, joint 
development of mission statements and accountability measures; at the state or federal level, policy 
changes or waivers to produce pooled funding streams for local programs or changes in 
regulations to eliminate major barriers to collaboration. However, much has been said about all 
these strategies being a means to an outcome and not an outcome of their own (Golden, 1991). 

♦The Healthy Start (Comprehensive Integrated School-Linked Services evaluation) initiative in 
California identified some of these indicators for collaborative functioning and change. 

Funding structures and/or practices: 

• Refinancing 

• Reallocation 

• Pooled funds 

• extracting and monitoring 

• Joint accountability 

Collaborative interaction: 

• Frequency and length of meetings 

• Formality of meetings 

• Meetings attendance 

• Level of participants in meetings 

• Procedure for setting agenda 
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• Conflict resolution procedures 

• Formal and informal communication methods among members 

• Decisionmaking practices 

• Agreement on vision 

Service provision: 

Staffing 

Use or nonuse of co-location 
Use of out-stationed woAers 
Changes in roles 
Use of cross-training 
Degree of service integration 
Referral procedures 

Interagency agreements about confidentiality 
Case management models 
Link of referrals to actual services 
New services 



PART III. INFORMATION SYSTEMS AND DATA INTEGRATION 

(Sources: Baker, 1993; CAPO, 1993; Goerge, 1991, 1993; Hebbeler, 1993; Stagner and Deutsch, 
1992; Urban Strategies Council, 1991; Usher, 1993; Wagner, 1993) 

As evidenced by recent legislative and privately funded initiatives and feasibility studies, there is 
now widespread agreement thai the development of complex information systems and interagency 
databases are crucial components of cunrent family-centered, integrated reform initiatives. [The 
Federal Administration Management Information System (FAMIS), the Adoption and Foster Care 
Analysis and Reporting System (AFCARS), the Pew Charitable Trusts' Children's Initiative ^ and 
the Annie E. Casey Foundation's Family to Family Initiative are just a few examples.] 

Several meetings are being convened to discuss these topics and the Ford Foundation sponsored 
two meetings (December 1991 and August 1992) to address issues of the potential of iriormation 
technology in human services. In addition, the Ford Foundation has awarded a grant to the 
National Center for Service Integration to survey the field and highlight ixmovative use of 
technology to enhance comprehensive services. 

Overview and definition of terms 

Data are raw, unorganized facts, figures, and observations that are meaningless individually; 
information is data that have been organized into a meaningful form. A system is a set of 
elements that operate together to accomplish organizational or personal objectives. 

Information systems are the procedures that are used to collect, record, store, retrieve, 
tabulate, and report data. The terras management information system (MIS), client information 
systems, and family information system are used interchangeably for information on children, 
families served, program operation, services, and costs. 

Federal and state agencies use various approaches to collect a range of data and organize it in 
different ways to accomplish different objectives. Dimensions on which information systems 
differ include: the level of governance (local, state, federal) for which the system is designed to 
provide information, kinds of data being collected, availability and type of computers available, 
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type of reports generated from the data, capability of the system to address the federal reporting 
requirements, and other factors. For monitoring purposes the data reported is on the populations 
being served, number of services being provided, ways in which the money is spent, and general 
compliance with requirements. 

Local programs use management information systems to track the delivery of services. For this 
use the systems can generate statistical reports on clients, produce lists of families in a variety of 
ways, track individual services, produce copies of family's records, provide links to information 
about other resources on the community, and so forth. 

A database is a collection of individual records that are related to one another. Each program and 
agency typically has their own database with related information. 

Data integration. The services integration movement was conceived against the backdrop of 
problems created by a categorical, fragmented system of service delivery that is not equipped to 
respond to the needs of low-income families. In the National Commission on Children's final 
report, the first recommendation to make programs and policies woik for children and their 
families is to "form an easily understood, comprehensive, and cohesive system that helps families 
seek and secure benefits and assistance, encourages collaboration among programs and services, 
and responds to the multiple needs and problems of severely troubled families" OSfational 
Commission on Children, 1993: 13), 

The call of the services integration movement suggests a need for the development of integrated 
data systems to facilitate and closely accompany the integration of services. 

Data integration refers to the linking or merging of data across agencies and across services. 
Because states have not had the experience of creating integrated databases, they usually don't have 
the personnel with the skills to do it and, therefore, they require a great deal of technical assistance 
to develop and implement integrated databases. 

Applications of data integration to systems reform. The following are some of the 
applications of integrated databases to the services integration movement 



• Complying with reporting requirements 

(Sources: Harbin, 1993; Hebbeler, 1993; Marks, 1993) 

Federal and state governments use the data from information systems for accountability purposes. 
Traditionally, the information systems have been used for reporting extensive data on program 
activity such as the number of services provided (e.g., how many families were visited, how many 
times)» expenditures, and whether eligibility for services was adequately documented. 

The following is an example of a legislative mandate to develop information systems that need to 
collect data from different systems. 

4Congress established the Program for Infants and Toddlers with Disabilities, (Part H of the 
Individuals with Disabilities Education Act ) to expand the availability of early intervention 
services. The law requires that each state develop a statewide comprehensive, coordinated, 
multidisciplinary, interagency program of early intervention services for infants and toddlers with 
disabilities and for their families. Fourteen components of the system are prescribed. The last of 
these components requires states to develop a data system that can report several Idnds of 
information about infants and toddlers in early intervention: the number in need of service, the 
number receiving service, and the type of service provided. Except for being able to supply these 
required data^ the parameters of the data system are left entirely up to the state. The particular 
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working of the data component suggests that Congress was interested in having information from 
states for program accoimtability purposes to see if the basic goals of the legislation were being met 
(Harbin, era/., 1993). 

Hebbeler's review (1993) of the eight states with extant data systems in early intervention found 
that states have used contrasting approaches to collect different kinds of data and to organize it in 
different ways. Most stale agencies are integrating information from different data systems to 
oversee service delivery by local programs, bill for services, and respond to the federal reporting 
requirements. Lead agencies are also sharing data with state legislatures to help them understand 
the need for early intervention. 



• Improving the delivery of services 
(Sources: CAPD, 1993; SRI, 1993; Wagner. 1993) 

In most current information systems, caseworkers can access only current service status in one 
program or department or, at best, the history of an individual's receipt of services from one 
program. To be more responsive to the needs of the family unit, as well as each family member, it 
may be that the service provider should know all the services that the family has received over time 
and those being received currently, and all of the diagnostic and eligibility labels or problems that 
the child or that family has experienced. Tracking the service provision by use of an interagency 
database provides case managers with family-level information from multiple agencies. 

The benefits of using integrated data for improved service delivery has not been fully accomplished 
because, so far, most of these data have not been made available to the service providers (despite 
the fact that, according to experts, it is technically possible to organize data at the individual or 
aggregate level in any state in which client-level data on services is being stored on computers). 
The two data-matching efforts described above are not yet being used to improve direct service 
delivery. 

♦The Services Reform Initiative (SRI) of Maryland's Office for Children, Youth, and Families 
developed HomeBase, currently one of the most advanced statewide case management information 
systems. HomeBase is specially designed for case managers providing family-focused, intensive 
case management services. The major components of the system (all of which are user-friendly) 
are: referrals, case management, ACTLOG, and reports. 

The referral section quickly captures key demographic information and indicators and transfers this 
information electronically to all SRI locations. (For example, in Prince George's County PCs have 
been set up in local Department of Social Services, Department of Juvenile Services, lead education 
agencies, and mental health offices where referrals arc made.) Woricers at the SRI locations input 
the required screens of irJformation and send them to a gatekeeper at the Department of Family 
Services. 

The case management section enhances this base of information while building the plan of care. 
It's also the place where case managers enroll providers, measure progress, and manage case 
closure. 

One of HomeBase's strongest assets is its ability to link information together. Individuals are 
associated with families and their needs are defmed according to Life Domain categories. All Life 
E>cmain categories arc associated with goals which are then linked to outcome measures as well as 
the strategies used to achieve these goals. 

ACTLOG captures the day-to-day activities of case managers, such as personal counseling, 
maldng phone calls, providing goods or services, and attending meetings. By graphicaUy 
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displaying this information, HomeBase is able to show case managers at a glance whether they are 
following the model advocated by their jurisdiction. One such model calls for intensive, home- 
based services that are concentrated in a six- week period. This information then becomes part of 
the self-evaliuition. ACTLOG produces printed reports and graphs for additional evaluation. 

Information in HomeBase is also used at the state level, where data is collected and compiled to 
give a broader view of the models being used throughout the state management process. 

♦The Pew Charitable Trusts' Children's Initiative has as primary goal of using similar kinds of 
management infomation systems to support family development practice, family-centered 
activities, streamlined intake and elic^biUty, and cross-agency case management and plaiming, 
including identification of child and tamily needs, service plans, service utilization, referrals, and 
service gaps and barriers. 

♦In California, a partnership of 12 private foundations is funding a large-scale effort to develop an 
integrated case management information system for the California Comprehensive Integrated 
School- Linked Services (Healthy Start) evaluation. This project is looldng at the most commonly 
used programs, developing a dictionary of common data elements, and writing the specifications 
for a software primarily designed to meet the needs of case managers working in the Healthy Start 
sites. Vendors will develop the software and sell iL Unfortunately, this information system will 
not be linked to any of the state's databases. 



Providing information for process and outcome evaluation 

The centralized data management systems that currentiy exist in most human service agencies do 
not provide policymakers and administrators with outcome data< An integrated, interagency 
database would provide information to support the process and outcome evaluation of system 
reform initiatives so that state and local officials and administrators have the critical information that 
they require to review the program effectiveness and efficiency and, based on the evidence, refine 
the program as necessary. Moreover, multi-site outcome evaluations depend on the successful 
implementation of a comprehensive database from which data can be extracted. 



Obtaining a broader understanding of families* experience in the system 

It is crucial to gather data from multiple agencies to conduct research that will lead to a better 
understanding of the experience of families and children. This information can help policymakers 
and administrators design more responsive services and policies. This type of research is 
infrequenUy done because there needs to be a single database, organized by individual and family 
units avail2d)le, with a complete history of information from aJQ previous extracts on all children and 
families in the database. 

In conducting research, interagency databases can be important resources to help increase the 
understanding of the experiences of populations being studied. For example, (i) many research 
questions and hypotheses about outcomes require using large samples of families. Small sample 
size prevents testing a large set of hypotheses because of an inability to include many variables in 
multivariate analyses, (ii) Low -incidence events in the context of large service systems (e.g., child 
deaths in institutions) are difficult to study in small samples, (iii) Larger samples allow the 
assessment of many possible deteiminants of outcomes. Researchers always prefer to include 
demographic data, individual or family problems, and service and community information in a 
multivariate analysis of a particular outcome (Goerge, 1993). Sampk surveys and the usual 
research methods cannot provide a view of the entire population. 




Interagency databases can also provide a longitudinal view of children's "human service careers*' 
from birth to adulthood. The importance of this for policy and practice, and what makes this type 
of research different is that the focus of the analysis becomes the child or the family longitudinally, 
and not the categorical service the child may receive. Longitudinal information about the sequence 
and duration of services that children receive through childhood and adolescence (i.e., make it 
possible to see what "door** is used by families entering the system) is rarely done. 

♦The Illinois statewide dau match is one of the only longitudinal efforts currently in place. 
Goerge (1991) describes the careers of children in child welfare using the common features of 
these careers, like the transitions among agencies, services, treatment tpodalities, and service 
settings, or the transitions between statuses or the occurrence of events other than moves among 
service settings. 

Underscoring the crucial importance of having a longitudinal perspective, Lynn Usher (1993 A) 
indicates that current child welfare information, gathered from looking at chUdien who are in the 
system at one point in time, indicates that the average length in foster care is 24-36 months. This 
information can be misleading. By analyzing the data longitudinally and tracking children's careers 
in two or three years, he found that 95 percent of children that were in emergency shelters and 
returned home immediately, never went back into foster care. These children are seldom counted 
in one-time, cross-sectional efforts. 

Similarly, the Department of Education follow-up studies have demonstrated that using one-time 
dropout data can overstate a portion of the problem because it does not determine whether dropouts 
eventually return to school. 



What does an integrated management information system look like? 

^Family Assistance Management Information System (FAMIS). In June 1980, Congress enacted 
Public Law 96-265 to provide federal fmancial support to state agencies as an incentive to develop 
and expand automated Title IV-A (AFDC) application processing and information retrieval 
systems. As a result, the Administration for Children and Families (ACF), Division of State 
System Approvals, developed the FAMIS as a general system design for AFDC. FAMIS is 
intended to assist the state in determining the basic requirements for systems design, augment the 
states* design definition, and establish basic standard systems components and file elements for 
future interfaces. 

With the passage of the Family Support Act, and its emphasis on coordinating all agencies 
providing income maintenance, social services, education, training, and employment programs, 
automat^ support became critical. ACF is strongly encouraging states to utilize their FAMIS 
systems in conjunction with a JOBS automated system (J AS) either by integrating their JAS in 
FAMIS, or interfacing with FAMIS. ChUd care services may also be included in the FAMIS 
system. Virtually all the systems developed with the enhanced funding include AFDC, Food 
Stamps, and Medicaid. Some also include the Child Support Enforcement Program. 

To curtail systems development costs and avoid duplication of effort, ACF has adopte<? a policy 
which requires states to transfer an existing FAMIS system from another state. This policy has 
eliminated some of the risks associated with developing a new system. Each tijne a system is 
transferred, both program and technical improvements are made to keep up with the ''state-of-the- 
art** 

Thirty stales currently meet FAMIS functional requirements, three arc developing plans to 
implement an automated statewide system, six are conducting some type of pilot implementation, 
and nine are in beginning planning stages. Only two states have no FAMIS activity. 
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♦Maryland's HomeBase system (see description above) establishes a common interface and relies 
upon a common set of data elements across agencies. This approach links caseworkers to relevant 
mformation and provides an alternative method for linking c ^^ting record systems, without 
incurring the cost of building an entirely new system. 

♦The Pew Charitable Trusts perceives the following "ideal" data integration system as the long- 
range goal for information systems development of the states selected to participate in the 
Children's Initiative. This goal would have to be reached by attaining a series of intermediate 
solutions developed and implemented over the life of their Children's Initiative. 

"A fuU'ScaU integrated, interagency database will be one that supports multiple applications, 
including cross-program streamlined eligibility processing and management of case data for the 
family-based services provided. All computeri^ information on service provision at the 
individual or family or program level would be accessible to all levels of staff, with access to 
information controlled by confidentiality , informed consent, and data sharing agreements between 
agencies. While the technical specifications of such systems are complex, given today's 
technology, it is possible (given sufficient political wiU and resources) to provide all levels of staff 
with the information that they need at their desktop, be it case-level or aggregate information" 
(CAPD, 1993: 6.1), 

Although no stale is close to being able to implement such a system, some efforts are now 
underway to use integrated information on children and families' needs, service utilization, and 
outcomes. 



Record linking or data matching 

This process involves pulling extracts (at a single point in time or on a periodic basis) from 
program files that exist in separate or unconnected, in-stale and local agencies' databases and 
matching children or families from each agency to create a single file. 

The following are examples of some of the efforts to integrate data by data matching. 

♦In California, the Urban Strategies Council and the Oakland Interagency Gr jp for School- 
Linked Services, a consortium of public health, education, and social service providers, conducted 
a one-time survey (data match) of the public service use of students attending eight schools in 
Oakland. The data match was mounted as part of their efforts to assess community needs and 
resources. The data match for this survey was obtained by electronically matching student records 
from eight schools with the databases of 19 income, health, and social service programs r\m by the 
city or county government Some of the findings of this d^ match were that (i) almost two out of 
three students in the study used public services, and large numbers were known to several 
programs at the same time; (ii) students who are having trouble in school arc also more likely io 
use public services; (iii) more resources are devoted to crisis responses than to prevention and 
maintenance supports; (iv) there are strong relationships between race, laiiguage, and service 
utilization; and (iv) mobility appeared to be associated with high program use and low school 
perfcMinance. 

♦Oakland^s data match was inspired by the New Beginnings project in San Diego, in which a 
comparable interagency group came together to collaborate in providing services to families around 
a low-incoroe» muiti-eihnic elementary school. Based on San EHego's data match fmdings, agency 
leadership determined that much greater coordination with schools was feasible. All this was part 
of the feasibility study which convinced agency partners that public agency investments, 
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particularly those of the county social services agency, justified collaboration in joint servdce 
delivery at the pilot school. 

♦A larger-scale (statewide) data match effort is being conducted by Robert Goerge, from the 
Chapin Hall Center for Children at the University of Chicago. Goerge developed an interagency 
database by linking administrative data on public child welfare and protective services, state mental 
health services, special education, juvenile correction services, AFDC and Medicaid programs, and 
public health (biith certificates). Because this database utilizes a series of departmental databa^ses, 
each of which contains all the children who received the particular services provided by a 
department, it is possible to survey the entire population of children in Illinois who arc receiving or 
have received those services over the course of a decade. This process has been complicated by 
the fact that the identification numbers across these disparate agencies are not assigned by one 
source or with one algorithm (such as the Social Security Number). This problem can be solved 
through "probabilistic" matching, using children's names and demographic characteristics. 



KEY QUESTIONS AND ISSUES ABOUT INFORMATION TECHNOLOGY 

(Sources: Baker, 1993; Duncan, etaL 1993; Goerge, 1993; Goldman, 1993; Hobbs, 1991; 
Joining Forces, et aL 1992; Marks, 1993; Stagner and Deutsch, 1992) 

Confidentiality. One of the barriers frequently identified in data sharing and data integration 
discussions are the regulations and statutes which protect the confidentiality of family and/or 
individual case records to the detriment of efficient and cost-effective service delivery. 
Confidentiality rules are a way to protect people from unwarranted invasions of their privacy and 
from use of information for a purpose for which it was never intended. The rules also protect 
people from the repercussions that could result because of negative public attitudes. 
Confidentiality restrictions are not limited to agencies. A number of professionals— doctors, 
mental health workers, and social workers— have legal obligations or strong ethical standards that 
prohibit the release of information about a patient, client, or student without consent 

Issues of confidentiality are also being addressed with changes on information in financial 
institutions (banks, credit agencies, etc.). 

The ACLU Project on Privacy and Technology (Goldman, 1993) argues that the uses of new 
technologies are always threatening to overtake current law, leaving society without a new set of 
laws or social mores to limit and define the uses of the new devices. But the solution is not to 
condemn automation and computerization, but to find ways to use them while protecting the rights 
to individual privacy. Keeping in mind these restrictions can help in understanding limits and 
exploring alternative approaches. 

Two recent comprehensive reports on managing confidential information in interagency projects 
(Hobbs, 1991; Joining Forces, et aL, 1992) found that, contrary to popular belief, the law is not 
tfie sole or even the principal barrier to efficient and effective exchange of confidential information. 
Therefore, their recommendations do not press for immediate major changes in the law, but focus 
on actions related to good program planning, improved automated file design, and more supportive 
administrative policies and practice. 

These reports suggest, among other things, that (i) interagency agreements greatly facilitate the 
exchange of information among service providing agencies. Shiing of data must be kept to a 
minimum, however, if agencies do not develop information -sharing agreements, (ii) Written 
parental consent will be necessary for complete information sharing to occur in specific 
circumstances, (iii) When the data is used for aggregate analysis only, names can be masked and 
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there is a lessened! risk of confidentiality problems. There are several barriers that have a major 
impact on daily operations and are management issues. 

Common idendfier and data elements. Since common identification numbers are not used 
by different departments, computer programs must be employed to pliably link children to families 
and records from one service population to another. Critical technical tasks in the construction of 
integrated databases are to insure that: (i) individuals arc linked to other.members of their families, 
so that data can be used at the family as well as individual level; (ii) and tiiat there are common 
identifiers or a set of identifiers across agencies (for example, that they all use the social security 
number), (iii) Multiple pieces of identifying information increase the probability of conea 
matches. Data must be stored in tiie same format and (v) data elements must be comparable 
(for example, that the definitions of ethnicity are the same) OBaker, 1993). 

Creating a ''user-friendly" environment The quality (and quantity) of data depends in 
great measure on how it is collected at the frontline* It is the extent Uiat data integration can be used 
to improve service delivery that will ultimately determine the worth of the data system. When data 
systems appear driven by the federal reporting mandates, by developers who arc unfamiliar with 
the service delivery system, or by the needs of outside evaluators, they decrease their value. 
Therefore, the single most important factor in designing a data system is to identify up front all the 
possible ways in which information can improve tiie provision of services. 

Typically, the information collected by the providers is fed to a centralized data processing unit 
where it is stored but seldom used. Frontline caseworkers and supervisors rarely receive useful 
information on their own clients or their own performance, and they have litde incentive to make 
sure that the data entered into the system is accurate. Other times, the infomiation system is only 
used to monitor the frontline worker, which they may perceive as a threat to their job. 

To increase staffs motivation to be efficient in data collection and recording, it is extremely 
important to provide casewoiters and their supervisors with the computer tools that may facilitate 
the collection and use of information. If computer systems help woiers do their jobs, they will 
become invested in the information that goes into them. The developers of the Head Start Family 
Information System, for example, are experimenting with using pen computers, ATM "type" 
cards, scaimers, and even small lap-top computers to facilitate the process of recording 
information. 

Caseworkers must also have significant input into the development of the forms to collect the data, 
the design of the evaluation, and the extensive technical assistance in coding. In addition, if people 
at the frondines and program administrators can use technology and reporting procedures which 
make their work easier to do or gain greater access to and ability to use information, it may lead to 
the collection of more accurate information and the development of data that more adequately 
describe circumstances and outcomes of clients. 

Size of database. Because comprehensive programs are multidimensional, an open question is 
how much data can or should be collected and maintained in interagency databases. Computer 
technology makes it possible to collect large amounts of data in new and creative ways. Many 
times the data collection process takes time and resources away from service delivery and imposes 
an added and unnecessary burden to families and frondine workers. Furthermore, programs 
usually have a wide range of data reporting requirements. When creating an information system, 
critical choices must be made about what data should be collected and analyzed and for what 
purposes. 

What comes first? Some experts believe tiiat lack of data integration is thwarting service 
integration. This group bdieves that without these integrated data, it is very difficult to design and 
practice in an integrated service environment A differing view is that services integration must 
come fu^t. When services integration becomes more prevalent, there will then be a greater demand 



27 



for integrated data, and systems will be developed to meet the demand. These different 
assumptions lead to varying views about the actions to be taken. 

Stagner and Deutsch (1992) consider a middle ground by thinking of integrated data— and the tools 
to use that data—as ways to reinforce efforts to improve the integration of services. Another way 
of looking at this is that developing linkages between agencies tl^ough data sharing will enhance 
their ability to "learn to collaborate." 

Categorical reporting requirements/accountability. Federal, state, and local agencies 
have different definitions and data structures for each of their categorical programs. Those become 
very important barriers to the integration of data at the local level. In many cases, local data 
systems are created to respond to federal monitoring requirements without thinking of how these 
systems might "fit" together. The process of filing separate reports is very time consuming. For 
example, one Settlement House in New York City reported having to file about 1007 separate 
fiscal reporting forms with city agencies over the course of the year (Marks, 1993). In addition, 
federal funds go to support development of incompatible computer systems. 

Tiirfism« Many of the agencies are protective of "then-" data. Turf wars between agencies and 
fear of political repercussions are some of the factors in these protective attitudes. Other times it is 
difficult to convince ageiKies to share data for integration at the casework or policymaking level or 
to gain access to centralized agency data for policy research or planning. In California, for 
example, the teen pregnancy program, health department, and school district have proprietary 
databases which do not "talk" to each other. 

Costs and funding. Integrated infomiation systems are a ^'public good.*" Each actor in the 
system benefits from having a system of integrated information, but it is not worth it for any 
individual to try to create such a system. Federal dollars usually cannot be used to create a system 
because costs must be charged direcdy to a categorical program. 

Resistance from line workers. There arc many reasons for the psychological resistances to 
change in working with automated information systems on the part of all the audiences. This is 
especially true about human services providers because they think that the dollars spent on 
evaluation and data collection would be better used to provide services. In addition, many social 
service providers do not think that what they do can be appropriately captured or have an 
unquestioned tenet that they have already helped by providing services, regardless of the 
measurable outcome. 

Lack of analysis expertise. During the last few years there have been promising 
developments in collecting family data. For example, the survey of income and program 
participation (SIPP) data; the development of a family data archive on CD-ROM; the second round 
of a national survey of families and households (NSF); and plans for a national health interview 
survey focusing on family health issues. As a result of these and previous efforts, large amounts 
of federal and stale data are available (21ill, 1991). However, there has not been a comparable 
effort in investment and capacity building for the analysis of these data. Most states lack the 
personnel with the level of expertise or the time to analyze and report on existing federal survey 
data. Several university centers and private firms arc beginning to help in the analysis of several 
components in existing national survey databases and/or provide technical assistance to states to do 
it For example, Child Trends helped design and analyze the survey for the National Commission 
on Children that focused on family functioning and family sQiesses. 
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PART IV. FEDERAL ROLE IN DATA SYSTEMS AND PROGRAM EVALUATION 

(Sources: Hendershoi and LeClerc, forthcoming; Institute for Research on Poverty, 1992; Levitan, 
1992; Zm and Daly, 1993) 

Although many serious problems related to federal involvement in data systems and program 
evaluation have been identified in this report, the federal government has a clear opportunity to play 
a uniquely powerful role in promoting improved and integrated data systems, and developing 
creative new methods of evaluation that fit the emerging child and family services revolution. 



Fragmentation of federal statistics 

At the national level, child and family-related statistics are collected by at least eight federal 
departments — HHS, HUD, Commerce, Labor, Education, Agriculture, Defense, and Justice. 
Within each department, several dozen offices and agencies are involved. They have very little 
comonunication with each other and are often unaware of the data and research being conducted by 
other federal offices. Most of these dau collection efforts are related to the needs of particular 
programs and collect data only about individuals and not about family units. Each office uses 
different definitions and measures. This fragmentation results in many serious problems, gaps, 
duplications, and missed opportunities to better understand the needs and problems of today's 
families (see Hendershot and LeClere, forthcoming; ZiU and Daly, 1993). 

Note: In 1988, Child Trends compiled a detailed guide to all federal child and family data. An 
updated version was published in October 1993 (see Zill and Daly, 1993). 

The basic administrative data systems through which the federal government requires the states to 
collect program information (e.g., AFDC, Food Stamps, Child Support) is similarly fragmented. 
The fact that different federal offices require so many different types of program data, at different 
times of the year, puts an enormous burden on state and local governments and agencies. These 
administrative data systems are not linked nor are they outcome focused. Some community-based 
programs which have several funding sources have to employ full-time staff simply to fill out the 
federal reporting forms. 

While several of these national data sources l-dve suffered funding cuts in recent years, there has 
been some progress. For example, we now have a couple of very important new longitudinal 
surveys — the Survey of Income and Program Participation, which collects information about 
multiple service use, and the National Survey of Families and Households. In addition, several 
government surveys are collecting information abopt the Hispanic population for the first time. 
However, the problems of coordination and integration remain serious. 



Federal Slole in Evaluation 

Although most evaluation research is performed by individuals and organizations in the private 
sector, the federal government is overwhelmingly the largest single fimder of evaluation studies. 
The federal government supports evaluation through requiiing and funding evaluation activity in 
most of the categorical programs, and through funding specific evaluation studies, especially of 
demonstration programs, through grants and contracts. 

However, it is important to note that several federal agencies themselves conduct audits and 
program evaluatioi\s. These range from the program evaluation reports of the General Accoimting 
Office and the HHS Inspector General's Office, the comprehensive reviews of the Office of 
Technology Assessment and the National Academy of Sciences, and the microsimulations and 
policy anSyses of the Congressional Budget Office (see Levitan, 1992). 
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Suggestions and Recommendations 



From our review of the literature and discussions with several experts we have compiled the 
following suggestions and recommendations for an enhanced and more effective federal role. 

• President Clinton should establish an Interagency Forum on Family-Related 
Statistics to coordinate existing federal data collection efforts and promote 
cooperative and collaborative activities between them and between the government 
and the private sector. A secondary focus of the Forum could be to pool efforts 
to synthesize and disseminate the findings of national surveys and research 
efforts in a timely fashion. Membership should draw from the eight federal 
departments (and dozens of offices) that sponsor child and family data collection 
and research and include representatives from relevant private sector 
organizations. 

This is a low-cost proposal, which is likely to receive support from the key Congressional 
committees, and is consistent with many of the National Performance Review report's 
recommendations to make government work better. There is also a precedent in the form of the 
Federal Interagency Forum on Aging Related Statistics which was established in 1986. This 
leconunendation was originally made by the Consortium of Family Organizations in 1991 after it 
conducted a series of interviews and meetings with federal statistics officials and others. It was 
then endorsed by participants in the Workshop on Family Data and Health Policy held in March 
1993 and sponsored by ^e National Center on Health Statistics and the Office of the Assistant 
Secretary for Planning and Evaluation in the Departmoit for Health and Human Services (see 
Hendei^hot and LeClere, 1993). A current study of this issue, funded by federal agencies, is 
underway and will report its recommendations by July 1994 (see page 31). 

•In establishing data and reporting requirements, the executive branch should 
specify and require the reporting of results (outcomes) and, in consultation with 
the states, establish realistic performance standards as a step towards achieving 
the outcomes. The government should place less emphasis on requiring reporting 
units of program activity. 

• Different programs and federal agencies should collaborate in agreeing on 
common elements, definitions, and timetables in their reporting requirements to 
make reporting less burdensome on states. 

• The federal government should develop requirements and/or incentives to states 
and localities to invest in improved and more integrated data systems, as the 
Administration for Children and Families has recently done in requiring states to 
use the FAMIS integrated data system if they wanted to receive federal dollars to 
expand their data systems. 

Since obtaining better data systems is in the national interest, and therefore a public good, it is 
reasonable for the federal government to pay for most, if not all, of the costs of doing so. The 
unanswered question is when multiple, categorical systems are involved, which federal office, 
usmg which source of funding, appropriated by which congressional committee, will have the 
incentive or the authority to require, or even encourage, increased integration of these systems. 

•It is generally a good idea for the federal government to mandate evaluation of 
new programs and policy initiatives, especially if the desired outcomes are clearly 
specified and attainable. However, such requirements must be realistic about 
timeframes and allocate sufficient dollars to do the studies properly. Importantly, 
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the legislation should not specify the detailed research designs to be used, but 
encourage states and local programs to be creative in designing approaches that fit 
their needs. 

• Some of the federal education and training monies in human services should be 
earmarked for training specialists in the new information technologies and 
evaluation methods. In addition, states should be encouraged and funded to 
provide each other with technical assistance. 

• The research and evaluation offices within federal agencies should collaborate 
with interested private foundations to convene experts to develop and disseminate 
a variety of descriptive inventories of new developments in information 
technology and best practices in data systems and evaluation, such as the 
development of user-friendly family assessments and other evaluation tools. 



PART V. EXAMPLES OF CURRENT RESEARCH, DATA INTEGRATION, AND EVALUATION 
INITIATIVES 

The following is a selection of current research, evaluation, and data system initiatives conducted at 
the national, stale, and local level, which focus on family-centered and integrated projects. Some 
of the organizations conducting these activities are also providing technical assistance to other new 
and emerging initiatives. 



NATIONAL LEVEL 

Coordination of Federal Child and Family Data Collection and Research 

In 1993, four federal agencies — the National Center for Health Statistics, Maternal and Child 
Health Bureau, Census Bureau, and the National Center for Education Statistics— contracted with 
consultant Martha Riche to identify and design a model for an interagency body that would help 
these federal statistical agencies work cooperatively on the identification, collection, and analysis of 
family data. Riche will begin the background interviews in January 1994, and her report and 
recommendations will be presented in June 1994. This effort builds on the work of tiie 
Consortium of Family Organizations' Family Data Project (see p. 30). Contact: Martha Riche, 
Population Reference Bureau, Inc., (202) 483-1100 or Gerry Hendershot, National Center for 
Health Statistics, (301) 436-7085. 

Community Based-Mental Health Services for Children in the Child Welfare 
System 

The office of the Assistant Secretary of Planning and Evaluation (ASPE) contracted with Macro 
International to conduct an assessment of die availability and adequacy of mental health services for 
children in the child welfare system who are not yet considered to be severely emotionally 
disturbed. The study assessed the four major systems that have an impact on emotionally 
disturbed children: child welfare, child mental health, juvenile justice, and schools. In addition, 
this descriptive study details the efforts of communities where child -serving agencies are working 
together and identifies the issues and barriers that these communities face in continuing to woik 
toward improved service delivery. Contact: Elisa Koff, ASPE/DHHS, HHH Building, Rm. 
404E. 200 Independence Ave. SW, Washington, DC 20201. (202) 690-5880. 



Comprehensive Child Development Program (CCDP) 

Twenty-four Comprehensive Child Etevelopment Programs (demonstration grants designed to 
encourage intensive, comprehensive, integrated services for infants and young children and their 
low-income families) were funded in fiscal years 1989- 1990. A second set of 10 CCDP projects 
were funded by the Administration on Children, Youth and Families (ACYF; in fiscal year 1992- 
1993. To meet the legislative mandate for an evaluation of the first group of CCDP projects, 
ACYF awarded two contracts: one to CSR, Incorporated for a feasibility and process evaluation 
and a second to Abt Associates for an impact evaluation. Abt Associates will also be conducting 
the process and impact evaluation of the second group of projects. 

For the impact evaluation, a randomized design was achieved in each site through the program 
requirement that the projects recruit more families than could be served, and then assign eligible 
families to program and comparison groups. Contact: Robert Ketterlinus, CSR, Incorporated, 
1400 Eye Street NW, Washington, DC 20005. (202) 842-7600. Or Robert St. Pierre, Abt 
Associates, 55 Wheeler Su-eet, Cambridge, MA 02138. (617) 492-7100. 

Community-Based Child Abuse and Neglect Prevention Programs 

In 1989 the National Center for Child Abuse and Neglect (NCCAN) funded nine Comprehensive 
Community-Based Child Abuse and Neglect Prevention programs to encourage community groups 
to develop community-based approaches to the prevention of child abuse and neglect. NCCAN 
contracted with CSR, Incorporated to design and implement a process and impact evaluation of 
these nine programs and to aid the programs in their efforts to design and implement their own 
internal program evaluations. The go^ of this third-party evaluation is to assess the 
comprehensive, community-based nature of the demonstration projects and evaluate the program 
components. Contact: Anne Baber-Kennedy, CSR, Incorporated, 1400 Eye Street NW, 
Washington, DC 20005. (202) 842-7600. 

Even Start Family Literacy Program 

Even Start is a demonstration program that offers educational services to both children and parents 
through an integrated program of early childhood education, adult basic skills training, and parent 
training. Four-year grants are offered to school districts, which provide the services direcUy or 
arrange for them through existing community programs. The evaluation of the Even Start program 
is a four-year national effort with three components; (1) construction of a large-scale database, the 
National Evaluation Information System; (2) an in-depth study of ten projects, half with 
randomized experimental designs, to complement the broad-based data with a detailed, small-scale 
analysis of the relationship between services received and short-term outcomes; and (3) other 
evaluation studies as desired by individual grantees. Contact: Robert St Pierre, Abt Associates, 
55 Wheeler Street, Cambridge, MA 02138. (617) 492-7100. 

Head Start Family Service Centers (FSC) 

FSCs arc demonstration projects designed to strengthen the capacity of Head Start programs to 
more effectively address the problems of substance abuse, employability, and low Uteracy, The 
goals of the national evaluation are to describe the process of implementing the FSCs and 
determine whether the families who participate in a Head Start FSC are more likely to identiJ[y and 
address the problems of substance abuse, employability, and low literacy, and, as a result, 
experience long-lasting benefits similar to benefits experienced by families who attend a regular 
Head Start program. 

Some of the Wave I and Wave 11 and all of the Wave lU grantee families are, or will be, randomly 
assigned to either the Head Start program or the Head Start program plus the FSC. To conduct the 
process evaluation, both FSC project directors and case managers complete questionnaires 
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regarding the irapleraentation, administration, and service delivery model of the FSC. For the 
impact evaluation, parents at the random assignment sites are interviewed in the fall and spring of 
the Head Start year and then in the spring of the following year regarding substance abuse 
treatment experiences, employment history, job training, and education. The final report will be 
completed in January 1996. Contact: Jean Lazyer, Abt Associates, 55 Wheeler Street, Cambridge, 
MA 02138. (617) 492-7100. 

Healiky Families America 

In partnership with the Ronald McDonald Children's Charities, the National Committee for 
Prevention of Child Abuse (NCPCA) launched the Healtiiy Families America effort to replicate 
Hawaii's intensive home visitor program across the country. In 1992, the National Center for 
Child Abuse and Neglect awarded NCPCA a contract to conduct a comprehensive evaluation of the 
initial and long-terra efficacy of this program. The study has three components: (1) a three-year 
controlled experiment of the intervention with families randomly assigned to treatment and control 
groups, with progress measured from intake to termination; (2) a follow-up assessment on a 
randomly selected group of families who terminated intensive services 12 to 18 months prior to the 
onset of this study; and (3) a six- month and twelve-month assessment of a smaller, randomly 
selected group of families identified through a hospital screening protocol or an in-person 
assessment interview as not being at risk. Contact: Deborah Daro, NCPCA, 332 S. Michigan 
Avenue, Suite 1600, Chicago, IL 60604. (312) 663-3520. 

Jobs Opportunities and Basic Skills Training Program (JOBS) 

A principal component of the Family Support Act, passed by Congress in 1988, is the Job 
Opportunities and Basic Skills Training program, or JOBS. JOBS is a program of education, 
training, and employment-related services to help AFDC parents move from welfare dependency to 
self-sufficiency. The Department of Health and Human Services (DHHS) commissioned die 
Manpower Demonstration Research Corporation (MDRC) to evaluate the effects of participation in 
the JOBS program on employment and welfare dependency. 

Child Trends, as a subcontractor to MDRC, is carrying out the study of the implications of the 
JOBS program for families with young children. This Child and Family Subgroup Study involves 
comprehensive assessments with AFDC families in which the youngest child is between three and 
five years of age when die parent enters the JOBS program. Interviews and child assessments are 
carried out in ^e homes of a portion of the full sample randomly assigned to one of the two 
experimental groups. In addition, two years after random assignment, respondents will be drawn 
from the experimental and control groups in three sites to be interviewed and further measures of 
the developmental status of the children will be obtained. The Department of Education plans to 
gather measures of academic progress of these children from school records and teacher surveys. 

The third study within the JOBS evaluation, the JOBS Observational Study, includes a subsample 
of the study of children and families and it involves videotaping mother-child interaction in tlie 
home. Contact: Kristin A. Moore, Child Trends, 4301 Connecticut Ave. NW, Suite 100, 
Washington, DC 20008. (202) 362-5580. 

Services Integration Pilot Projects 

In 1985, DHHS awarded five multi-year Services Integration Pilot Projects (SEP?) to Arizona, 
Rorida, Maine, Oklahoma, and South Carolina to develop plans to integrate services to assist 
varied segments of their populations. In 1991 , James Bell Associates began a retrospective 
evaluation of the states efforts, involving an analysis of the states' self-evalu&tion reports, initial 
proposals, and other project documents, as well as site visits to each of the projects. Contact: 
Elyse Kaye, James Bell Associates, 2200 Clarendon Boulevard, Suite 1005, Arlington, VA 
22201. (703) 908-9725. 



Youth At Risk: Definitions^ Prevalence and Approaches to Service Delivery 



ASPE contracted with the Urban Institute to examine comprehensive service integration programs 
for adolescents 10-15. The overall goals of this descriptive study were to: (1) documenl how 
comprehensive, integrated services are delivered to at-risk youth at five locations; (2) identify 
barrbrs to providing comprehensive services, and means for facilitating services integration for at- 
risk youth; (3) examine the role of federal, state, and local government and the nonprofit sector in 
impeding or facilitating service integration for at-risk youth; and (4) identify issues for further 
research on the provision of comprehensive and/or integrated services for at-risk youth. Contact: 
Martha Burt, Urban Institute, 2100 M Street NW, Washington, DC 20037. (202) 833-7200. 

Study of Management Information Systems for Comprehensive Service Initiatives 

The National Center for Service Integration, with funding from the Ford Foundation, will conduct 
a study on the information system needs of collaborative initiatives, and the technologies currently 
available to meet those needs. This project will have three products: (1) a description and analysis 
of the requirements which a management information system (MIS) (for comprehensive services 
initiatives) should meet; (2) a framework characterizing the current service integration management 
information systems; and (3) a report which will assess the current supply of MIS in view of 
identified requirements, identify gaps in technology, and list decision criteria and political, 
technical, and training issues for initiatives to consider as they select, design, and implement 
management information systems to support comprehensive, integrated services for children and 
families. Contact: Deborah Both, National (center for Service Integration, 5111 Pike, Falls 
Church, VA 22041. (703) 824-0800, 



STATE 

California Statewide Evaluation of Comprehensive Integrated School-Linked 
Services (CISLS) 

In June 1992, 40 sites throughout California were granted funds, under the legislative authority of 
SB 620, to begin a three-year effort to restructure education, healdi, and soci^ services systems in 
their communities so that children and families would have readier access to a broader array of 
services through the schools. SB 620 also mandated that both a statewide evaluation and 
individual, local evaluations be conducted. The Foundation Consortium for School-Linked 
Services, in partnership with the State of California, contracted with SRI International to evaluate 
this initiative. 

The CISLS evaluation employs a longitudinal design that compares several aspects of both the 
{^-ocess and outcomes of the initiative to a baseline measurement of conditions that prevailed when 
it began. Repeated measures of process characteristics, services provided, and outcomes will be 
taken to yield a picture of the evolution of the initiative over the three years of the evaluation. 
Contact: Mary Wagner, SRI International, 333 Ravenswood Avenue, Menlo Park, CA 94025- 
3483. (415) 326-6200. 

CGPA Policy Academy on Families and Children At Risk 

The Policy Academy on Families and Children At Risk, conducted by the Council of Governors' 
Policy Advisors (CGPA), sought to help states focus on the multiple problems confronting 
children and their families across the country. The Family Academy brought together, in an 
intensive process, ir^teragency teams of top-level policymakers from several states. The 
independent evaluati on of the Family Academy was conducted by Policy Research Associates, Inc. 
Using several appioaches, including surveys and interviews, the evaluation sought to assess the 



effectiveness of the Academy process for bringing about a change, document the outcomes for the 
participating states, and identify key lessons learned. Contact: Joseph J. Cocozza, Policy Research 
Associates, Inc., 262 E)elaware Avenue, Delmar, NY 12054. (518) 439-7415. 

Evaluation of the Services Reform Initiative Maryland 

Prince George's County and the State of Maryland embarked on the Services Reform Iniditive 
(SRI) to achieve fundamental changes in the planning and delivery of services to families and 
children, A goal of SRI is to develop a noncategorical system of services that responds 
appn^riately to the needs of each family and provides success to the proper level and mix of 
services. A team from the Research Triangle Institute (RTI) and the School of Social Work at the 
University of North Carolina a: Chapel Hill conducted an evaluability assessment of the SRI. This 
team recommended that each state and local governance entity measure a set of outcomes, starting 
with individual families and progressing to the entire community, according to specific phases of 
development of the Initiative. To strengthen its capacity for self-evaluation, the Prince (jeorge's 
Commission for Families decided to conduct an analysis of the families being served. Based on 
the fmdings of that analysis, stakeholders txx)k several steps to bring services in line with a 
different model of intensive family preservation services. In addition, SRI instituted a process to 
monitor and evaluate the targeting and delivery of family preservation services. Contact Charles 
Lynn Usher, School of Social Work, The University of North Carolina, Chapel Hill, NC 27599- 
3550.(919) 962-1225. 

Family Preservation Integration Projects for High-Risk School Age Children and 
Their Families In Minnesota 

In 1991, the Minnesota Department of Human Services provided funds to develop case studies of 
strong family preservation projects operating in several Minnesota coimties. The goal of this 
descriptive study was to have a clearer understanding of the way in which local agencies 
collaborate to help families maintain a nurturing home and avoid out-of-home placement for their 
children, and how school-age children are helped to remain in their homes and communities. 
Contact: Esther Watienberg, Center for Urban and Regional Affairs, 330 Hubert Humphrey 
Center, 301 19th Avenue South. Minneapolis, MN 55455. (612) 625-1551. 

Family Preservation Services in New Jersey 

The New Jersey Family Preservation Program is modeled after the Homebuilders intensive 
placement prevention program. The evaluation of this initiative was designed to address one of the 
. perceived gaps in previous evaluations: the minimal use of control groups to determine the net 
effect of the Homebuilders' intervention. In addition, an attempt was made during this evaluation 
to understand the efficacy of the family preservation intervention in the context of the ecology of 
each family treated. Contact: Leonard Feldman, Bureau of Research, Evaluation, and Quality 
Assurance, New Jersey Division of Youth and Family Services, 50 E. State Street, CN 717, 
Trenton, NJ 08625-0717, (609) 292-8510. 

Illinois Family First Placement Prevention Program 

The Illinois Family First placement prevention program was design to prevent out-of-home 
placement of abused and neglected children. In 1989, the Illinois Department of Children and 
Family Seiviocs (DCFS) contracted with the Chapin Hall Center for Children to evaluate the 
program. The evaluation was designed to test the effects of this program on out-of-home 
placement of children and other outcomes, such as subsequent child maltreatment, the length of 
time families remain in the public child welfare system, and several measures of child and family 
well-being. This study included the largest randomized experiment conducted to date on a family 
preservation program. Qualitative and quantitative data were gathered from multiple sources to 
describe the program, workers who provided services, types and amounts of services provided. 
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families served, and case outcomes. Contact: John Schuennan, Chapin Hall Center for Children, 
The University of Chicago, 1 155 East 60th Street, Chicago, IL 60637. (312) 753-7922. 

Michigan's Families First 

The Michigan Department of Social Services contracted with University Associates of Lansing, 
Michigan, to conduct the evaluation of the Families First program. Families First is designed to 
keep children who would otherwise be placed in out-of-horae care in the child welfare, juvenile 
justice, and mental healtii systems safely witii tiieir families. The purpose of the evaluation is to 
determine the effectiveness of the model and to compare its cost and case outcomes with those of 
foster care placement Contact: Karen Smith, Office of Communications, 235 S. Grand Avenue, 
Suite 1510, Lansing, MI 48909. (517) 373-7394. 

MulU'Stau Foster Care Data Archive 

Ch^in Hall Center for Children is establishing a multi-state archive of foster care data constructed 
from computerized case records collected by state agencies (in California, Illinois, Michigan, New 
Jersey, New York, and Texas) to track children living in child welfare settings. Two more states 
will be added during each of the next four years. This will result in an archive containing daia 
from about 70% of the U.S. foster care population. Design principles emphasize data features that 
promote comparative (between state) and longitudinal (over time) analysis capabilities. Application 
of the archive will be extended by developing integrated data sets diat link case data from other 
service agencies and by developing geographical techniques that allow "community context" to be 
brought into foster care analysis. Contact: Robert Goeigc, Chapin Hall Center for Children, The 
University of Chicago, 5801 Soudi EUis, Chicago, IL 60673. (312) 753-5900. 

Pew Charitable Trusts Children's Initiative 

The Pew Children's Initiative seeks to improve the health and weU-being of children on a broad 
scale in three states over an eleven-year period (which includes a thirty-month planning phase and 
nine-year implementation). It sets forth critical outcomes (improved child health, child 
development, and family functioning, and a reduction in barriers to adequate school performance) 
and caUs for system changes necessary to achieve these outcomes. The nature of the Initiative 
poses very difficult design issues for the evaluation of impacts on outcomes for children and 
families. After consultation with a variety of experts, Pew decided that the evaluation vidll measure 
and track changes in service utilization and outcomes for children and families within communities 
and subgroups, comparing indicators before and after implementation of the Initiati\'e. Because 
these prc-post measures of charige cannot be used to detemiine whether the Initiative was the 
critic^ factor in bringing the observed changes, Pew is contemplating the possibility of carrying 
out an impact assessment of one or two key components of the service strategy in the later 
expansion phases, after project start-up and once there is evidence of successful implementation. 
Contact: Patti Pacreasi, Pew Charitable Trusts, 1 Commerce Square, 2005 Market Street, Suite 
1700, PhUadelphia, PA 19103. (215) 508-3330. 

Program Coordination in Texas Welfare-Employment Programs 

The Texas Legislature directed the Texas Etepartment of Commerce—the state Job Training 
Partnership Act (JTPA) entity— to contract with the University of Texas at Austin for an evaluation 
of public and private coordination efforts affecting the delivery of training end related services to 
ATOC recipients, especially those targeted for services under the new JOBS program. The 
evaluation documented the nature and extent of coordination among Texas welfarc-to-work 
programs at the state and local level in the pre- JOBS period and measured its effects on participant 
characteristics, program treatments, and inunediate and longer-terra outcomes. Contact: 
Christopher T. King, Center for the Study of Human Resources, L.B. J. School of Public Affairs, 
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The University of Texas ai Austin, Drawer Y, University Station, Austin, TX 787 13-7450. (512) 
471-4962. 



CITY/LOCAL 

Facilitation of Community'^Based Service Integration 

In June 1991 , ASPE provided funds to support slate and local service integration initiatives for a 
National Service Integration Resource Center and planning grants to as many as six organizations 
to act as facilitators of local integration initiatives. In January 1993, James Bell Associates was 
awarded a contract to study the six service integration planning grants. The puipose of this 
evaluation is to identify the lessons from the strategies employed, barriers encountered, and 
approaches to addressing these barriers. Contact: Elyse Kaye, James Bell Associates, 2200 
Clarendon Boulevard, Suite 1005, Arlington, VA 22201. (703) 908-9725. 

New Beginnings 

New Beginnings is a collaborative initiative involving executives from the San Diego County 
Department of Social Services and Health and Probation, the City of San Diego, the Community 
College District, and the city Housing Commission, Parks and Recreation Department, and library. 
Comprehensive services for low-income families and children are provided at the Hamilton 
Elementary School in San Diego, 

The objectives of the evaluation of New Beginnings are to document, describe, and assess the 
implementation of New Beginnings and assess outcomes of the system and services for children 
and families. The process evaluation employs three approaches to understanding the 
implementation of New Beginnings. A fidelity approach assesses the extent to which the New 
Beginnings design has been implemented as planned and how various parts of the program have fit 
togetl»er. A process approach focuses on the operation of the system thus far, recording events, 
behaviors of participants, and how participants perceive the system. And a system ^proach 
identifies and measures what features of the system are associated with positive outcomes for 
families and children. To assess the effectiveness of the New Beginnings services and measure the 
outcomes for families, the evaluation tracked students at the program site (Hamilton school) and at 
a comparison school. Contact: Larry Guthrie, Director, Student At Risk Program, Far West 
Laboratory, 730 Harrison Street, San Francisco, CA 94107-1242. (415) 565-3012. 

Oakland Partnership for Change. Linking Schools, Services, and Communities to 
Serve Oakland Youth 

The Partnership for Change is a data match project conducted by the Urban Strategics Council and 
the Oakland Interagency Group for School-Lirited Services, a consortium of pubhc health, 
education, and social service providers. The data for the stuvey was obtained by electronically 
matching student records from eight schools with the databases of 19 income, health, and social 
service programs, primarily those run by city or county government To protect confidentiality, all 
personal identifiers were removed before turning the data over for analysis. Contact: Susan 
Lubeck, Urban Strategies CouncU, 672 13th Street, Suite 200, Oakland, CA 94612. (510) 893- 
6657. 

Ramsey County, Minnesota. Community Human Services Department 

The Office of Research and Evaluation of the Ramsey County Community Human Services 
Department has developed a service- wide approach to evaluating the effectiveness of the mental 
hei th and social services provided by County staff or purchased tlirough conu^ts. The Office 
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integrates infonnation gathered through program evaluation into the decisionmaking processes of 
staff, management, the County Board, and other interested citizens. Program evaluation is 
undertaken to: (1) improve program performance; (2) infomi planning and funding decisions; and 
(3) establish accountability for social and mental health services. Contact: Charles Gene Lyle, 
Office of Research and Evaluation, Ramsey County Community Human Services Department, St 
Paul, MN 55101. (612) 298-5351. 

Turning Points 

The Turning Points program provides comprehensive and ongoing support to children attending 
selected schools in the District of Columbia and to their famihes. This support is provided in an 
effort to prevent family dysfunction and delinquency, foster development of healthy and productive 
lifestyles, orient adolescents to a lifetime of meaningful work, and encourage and support good 
citizenship. With funding from ASPE. James Bell and Associates developed a comprehensive 
evaluation study design plan for a multi-site, junior high school-based comprehensive service 
program sponsored by the DC Department of Human Services. The evaluation study has three 
components; process, cost-effectiveness, and outcome. Contact* Sonia Chessin, James Bell and 
Associates, Inc., 2200 Clarendon Boulevard, Suite 1005, Arlington, VA 22201. (703) 908-9725. 

Walbridge Caring Communities Program 

The Caring Communities model was developed in Missouri through the collaborative efforts of the 
Departments of Health, Mental Health, Social Services, and Elementary and Secondary Education. 
The goal of the collaboration was to create a total restmcturing of the way services are offered to 
children and families in need. The Caring Communities programs are designed to overcome 
barriers such as cultural insensitivity, bureaucratic barriers, remote locations, outsiders doing 
things ''to** communities rather than "with" them, narrow categorical approaches to problems, and 
failure to recognize the family as the appropriate unit of service. The evaluation of diese programs 
includes a formative evaluation which describes the process by which each project is created, 
services currently offered, and the short-term achievem^ts of each project in delivering services. 
It also includes an outcome evaluation with data from the school system (academic progress among 
children served), the Department of Social Services (data on child abuse/neglect reports), and the 
Department of Justice (involvement with the juvenile justice system). Contact: Khatib Waheed, 
Director, Walbridge Caring Communities Program, 5019 Alcott, St Louis, MO 03120. (314) 261- 
8282. 



GUIDES 



Evaluation 

Building Strong Foundations: Evaluation Strategies for Family Resource Programs. Family 
Resource Coalition, 1993. FRC, 230 North Michigan Avenue, Suite 1625, Chicago, IL 60601. 
(312) 726-4750. 

Building Villages to Raise Our Children: Evaluation. Pamela Miller, Harvard Family Research 
Project, 1993. Harvard Graduate School of Education, Longfellow Hall, Appian Way, 
Cambridge, MA 02138. (617) 495-9108. 

Charting Change in Infants, Families and Services. A Guide to Program Evaluation for 
Administrators and Practitioners. National Center for Clinical Infant Programs, 1987. NCCIP. 
2000 14th Street North, Suite 380, Arlington, VA 22201. (703) 528-4300. 

Evaluating Child Abuse Prevention Programs. Ellen Gray and Joan Di Leonardi, National 
Committee for the Prevention of Child Abuse, 1987. NCPCA, 332 South Michigan Avenue, Suite 
950, Chicago, IL 60604^357. (312) 663-3520. 

The Challenges of Evaluating State Family Support and Education Initiatives. An Evaluation 
Framework Heather Weiss and Robert Halpem. Harvard Family Research Project, 1989. 

Random Assignment 

Videotape produced by Abt Associates, 55 Wheeler Street, Cambridge, MA 02138. (617) 492- 
7100. 

Cost Effectiveness 

Cost-Effectiveness: A Primer. H. Levin. Beverly Hills, CA: Sage Publications, 1983. P.O. Box 
5084, Newbury Pait, CA 91359-9924. (805) 499-0721. 

Data Integration 

Data Match: A Tool for Assessment and Advocacy. Urban Strategies Council, 672 13th Street, 
#200, Oakland, CA 94612. This guide is designed to assist communities who are thinking about 
beginning a data match. It describes the purposes of the match and lays out the steps involved in 
acquiring and then matching the needed data. It discusses how to handle issues of confidentiality. 

Confidentiality 

Confidentiality and Collaboration: Information Sharing in Interagency Efforts. Joining Forces; 
American Public Welfare Association; Center for Law and Social Policy; Council of Chief State 
School Officers, and Education Commission of the States. CCSSO, One Massachusetts Avenue, 
Suite 700, Washington, DC 20001. 

Tackling 0ie Confidentiality Barrier: A Practical Guide for Integrated Family Services. L. Hobbs, 
San Diego, CA; New Beginnings, 1991. 
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